[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:12:03 dist-test-slave-v5j9 krb5kdc[30640](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:12:03 dist-test-slave-v5j9 krb5kdc[30640](info): set up 2 sockets
Sep 24 16:12:03 dist-test-slave-v5j9 krb5kdc[30640](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260924 16:12:04.390352 30632 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Sep 24 16:12:04 dist-test-slave-v5j9 krb5kdc[30640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:12:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:12:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.234.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab.
Entry for principal kudu/127.29.234.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab.
I20260924 16:12:04.567310 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:35839
--webserver_interface=127.29.234.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab
--principal=kudu/127.29.234.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:39439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.234.62:35839
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:12:04.987924 30653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:04.988293 30653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:04.988404 30653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:04.996488 30653 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:12:04.996616 30653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:04.996665 30653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:04.996716 30653 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:12:04.996759 30653 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:12:05.007282 30653 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:39439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.234.62:35839
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab
--principal=kudu/127.29.234.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:35839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.234.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:12:05.008859 30653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:05.010609 30653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:05.033236 30659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:05.033257 30661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:05.033573 30658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:05.034709 30653 server_base.cc:1034] running on GCE node
I20260924 16:12:05.035835 30653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:05.038043 30653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:05.039315 30653 hybrid_clock.cc:648] HybridClock initialized: now 1790266325039259 us; error 39 us; skew 500 ppm
I20260924 16:12:05.039768 30653 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:12:05 dist-test-slave-v5j9 krb5kdc[30640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266325, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:05.043520 30653 init.cc:377] Logged in from keytab as kudu/127.29.234.62@KRBTEST.COM (short username kudu)
I20260924 16:12:05.045632 30653 webserver.cc:460] Webserver started at http://127.29.234.62:39559/ using document root <none> and password file <none>
I20260924 16:12:05.046396 30653 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:05.046540 30653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:05.046862 30653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:05.050375 30653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/minicluster-data/master-0/data/instance:
uuid: "4b98071bedac47c4999b390f280dae4b"
format_stamp: "Formatted at 2026-09-24 16:12:05 on dist-test-slave-v5j9"
server_key: "f40d56fac78a277952bcde105052b652"
server_key_iv: "3957f83322ecd2b753f511e72bdc31ba"
server_key_version: "encryptionkey@0"
I20260924 16:12:05.051375 30653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/minicluster-data/master-0/wal/instance:
uuid: "4b98071bedac47c4999b390f280dae4b"
format_stamp: "Formatted at 2026-09-24 16:12:05 on dist-test-slave-v5j9"
server_key: "f40d56fac78a277952bcde105052b652"
server_key_iv: "3957f83322ecd2b753f511e72bdc31ba"
server_key_version: "encryptionkey@0"
I20260924 16:12:05.058017 30653 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260924 16:12:05.062294 30668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:05.063822 30653 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260924 16:12:05.064038 30653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/minicluster-data/master-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/minicluster-data/master-0/wal
uuid: "4b98071bedac47c4999b390f280dae4b"
format_stamp: "Formatted at 2026-09-24 16:12:05 on dist-test-slave-v5j9"
server_key: "f40d56fac78a277952bcde105052b652"
server_key_iv: "3957f83322ecd2b753f511e72bdc31ba"
server_key_version: "encryptionkey@0"
I20260924 16:12:05.064276 30653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:05.082304 30653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:05.086844 30653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:05.087143 30653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:05.107295 30653 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.62:35839
I20260924 16:12:05.107342 30720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.62:35839 every 8 connection(s)
I20260924 16:12:05.108798 30653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
I20260924 16:12:05.113040 30721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:05.117803 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 30653
I20260924 16:12:05.118211 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/minicluster-data/master-0/wal/instance
I20260924 16:12:05.120812 30632 external_mini_cluster.cc:1428] Setting key de277cd0eda00d537896f43a7a789c78
I20260924 16:12:05.126588 30721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b98071bedac47c4999b390f280dae4b: Bootstrap starting.
I20260924 16:12:05.130093 30721 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b98071bedac47c4999b390f280dae4b: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:05.131152 30721 log.cc:824] T 00000000000000000000000000000000 P 4b98071bedac47c4999b390f280dae4b: Log is configured to *not* fsync() on all Append() calls
I20260924 16:12:05.134312 30721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b98071bedac47c4999b390f280dae4b: No bootstrap required, opened a new log
I20260924 16:12:05.144492 30721 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b98071bedac47c4999b390f280dae4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b98071bedac47c4999b390f280dae4b" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 35839 } }
I20260924 16:12:05.145079 30721 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b98071bedac47c4999b390f280dae4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:05.145198 30721 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b98071bedac47c4999b390f280dae4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b98071bedac47c4999b390f280dae4b, State: Initialized, Role: FOLLOWER
I20260924 16:12:05.146008 30721 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b98071bedac47c4999b390f280dae4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b98071bedac47c4999b390f280dae4b" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 35839 } }
I20260924 16:12:05.146308 30721 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b98071bedac47c4999b390f280dae4b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:12:05.146467 30721 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b98071bedac47c4999b390f280dae4b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:12:05.146711 30721 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b98071bedac47c4999b390f280dae4b [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:05.150097 30721 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b98071bedac47c4999b390f280dae4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b98071bedac47c4999b390f280dae4b" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 35839 } }
I20260924 16:12:05.150689 30721 leader_election.cc:302] T 00000000000000000000000000000000 P 4b98071bedac47c4999b390f280dae4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b98071bedac47c4999b390f280dae4b; no voters:
I20260924 16:12:05.151377 30721 leader_election.cc:288] T 00000000000000000000000000000000 P 4b98071bedac47c4999b390f280dae4b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:12:05.151530 30726 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b98071bedac47c4999b390f280dae4b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:12:05.153648 30726 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b98071bedac47c4999b390f280dae4b [term 1 LEADER]: Becoming Leader. State: Replica: 4b98071bedac47c4999b390f280dae4b, State: Running, Role: LEADER
I20260924 16:12:05.154088 30721 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b98071bedac47c4999b390f280dae4b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:12:05.154356 30726 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b98071bedac47c4999b390f280dae4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b98071bedac47c4999b390f280dae4b" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 35839 } }
Sep 24 16:12:05 dist-test-slave-v5j9 krb5kdc[30640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:12:05.157943 30727 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b98071bedac47c4999b390f280dae4b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b98071bedac47c4999b390f280dae4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b98071bedac47c4999b390f280dae4b" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 35839 } } }
I20260924 16:12:05.158311 30727 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b98071bedac47c4999b390f280dae4b [sys.catalog]: This master's current role is: LEADER
I20260924 16:12:05.167304 30732 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:12:05.167245 30728 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b98071bedac47c4999b390f280dae4b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b98071bedac47c4999b390f280dae4b. Latest consensus state: current_term: 1 leader_uuid: "4b98071bedac47c4999b390f280dae4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b98071bedac47c4999b390f280dae4b" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 35839 } } }
I20260924 16:12:05.167611 30728 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b98071bedac47c4999b390f280dae4b [sys.catalog]: This master's current role is: LEADER
I20260924 16:12:05.172596 30732 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:12:05.183519 30723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:05.126079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59610 (local address 127.29.234.62:35839)
0924 16:12:05.127021 (+ 942us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:05.127036 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:05.128577 (+ 1541us) server_negotiation.cc:407] Connection header received
0924 16:12:05.139970 (+ 11393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:05.140003 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:05.140667 (+ 664us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:05.141177 (+ 510us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:05.144259 (+ 3082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:05.146318 (+ 2059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:05.148237 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:05.149581 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:05.149810 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:05.158283 (+ 8473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:05.158304 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:05.158335 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:05.158414 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:05.177394 (+ 18980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:05.178719 (+ 1325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:05.178728 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:05.178737 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:05.178942 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:05.179535 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:05.179544 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:05.179548 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:05.180144 (+ 596us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:05.180521 (+ 377us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:05.181596 (+ 1075us) server_negotiation.cc:299] Negotiation successful
0924 16:12:05.182042 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":691,"thread_start_us":402,"threads_started":1}
I20260924 16:12:05.188098 30732 catalog_manager.cc:1347] Generated new cluster ID: d442463b724f4cb7988b4fc5832b218a
I20260924 16:12:05.188265 30732 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:12:05.211622 30732 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:12:05.213485 30732 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:12:05.221256 30732 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b98071bedac47c4999b390f280dae4b: Generated new TSK 0
I20260924 16:12:05.222250 30732 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.234.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab.
Entry for principal kudu/127.29.234.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab.
I20260924 16:12:05.273084 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.1:0
--local_ip_for_outbound_sockets=127.29.234.1
--webserver_interface=127.29.234.1
--webserver_port=0
--tserver_master_addrs=127.29.234.62:35839
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab
--principal=kudu/127.29.234.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:39439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:12:05.628866 30747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:05.629199 30747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:05.629406 30747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:05.636652 30747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:05.636816 30747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:05.637004 30747 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.1
I20260924 16:12:05.646929 30747 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:39439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab
--principal=kudu/127.29.234.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.234.1
--webserver_port=0
--tserver_master_addrs=127.29.234.62:35839
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.1
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:12:05.648295 30747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:05.649817 30747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:05.663885 30755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:05.664019 30752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:05.664220 30753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:05.664963 30747 server_base.cc:1034] running on GCE node
I20260924 16:12:05.665603 30747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:05.666738 30747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:05.667972 30747 hybrid_clock.cc:648] HybridClock initialized: now 1790266325667902 us; error 61 us; skew 500 ppm
I20260924 16:12:05.668439 30747 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:12:05 dist-test-slave-v5j9 krb5kdc[30640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266325, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:05.672001 30747 init.cc:377] Logged in from keytab as kudu/127.29.234.1@KRBTEST.COM (short username kudu)
I20260924 16:12:05.673710 30747 webserver.cc:460] Webserver started at http://127.29.234.1:44669/ using document root <none> and password file <none>
I20260924 16:12:05.674465 30747 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:05.674618 30747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:05.674958 30747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:05.678063 30747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/minicluster-data/ts-0/data/instance:
uuid: "806828369087427eaa69c842f3503354"
format_stamp: "Formatted at 2026-09-24 16:12:05 on dist-test-slave-v5j9"
server_key: "981b96c7314dc81879ade393698ffda8"
server_key_iv: "a2c04292d0cb26aa35d3c4cd2d380bb3"
server_key_version: "encryptionkey@0"
I20260924 16:12:05.678889 30747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/minicluster-data/ts-0/wal/instance:
uuid: "806828369087427eaa69c842f3503354"
format_stamp: "Formatted at 2026-09-24 16:12:05 on dist-test-slave-v5j9"
server_key: "981b96c7314dc81879ade393698ffda8"
server_key_iv: "a2c04292d0cb26aa35d3c4cd2d380bb3"
server_key_version: "encryptionkey@0"
I20260924 16:12:05.684753 30747 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.006s
I20260924 16:12:05.688987 30763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:05.690716 30747 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:12:05.690902 30747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/minicluster-data/ts-0/wal
uuid: "806828369087427eaa69c842f3503354"
format_stamp: "Formatted at 2026-09-24 16:12:05 on dist-test-slave-v5j9"
server_key: "981b96c7314dc81879ade393698ffda8"
server_key_iv: "a2c04292d0cb26aa35d3c4cd2d380bb3"
server_key_version: "encryptionkey@0"
I20260924 16:12:05.691118 30747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:05.705596 30747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:05.709966 30747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:05.710281 30747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:05.711444 30747 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:12:05.712891 30747 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:12:05.712998 30747 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:05.713155 30747 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:12:05.713238 30747 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:05.749179 30747 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.1:38815
I20260924 16:12:05.749219 30878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.1:38815 every 8 connection(s)
I20260924 16:12:05.750707 30747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
I20260924 16:12:05.751775 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 30747
I20260924 16:12:05.752013 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/minicluster-data/ts-0/wal/instance
I20260924 16:12:05.752692 30632 external_mini_cluster.cc:1428] Setting key b231bced1b67e2325387c9b943a5d782
Sep 24 16:12:05 dist-test-slave-v5j9 krb5kdc[30640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266325, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.1@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.234.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.2@KRBTEST.COM" created.
I20260924 16:12:05.773325 30881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:05.753813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.1:58565 (local address 127.29.234.62:35839)
0924 16:12:05.754407 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:05.754412 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:05.755787 (+ 1375us) server_negotiation.cc:407] Connection header received
0924 16:12:05.757836 (+ 2049us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:05.757847 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:05.757940 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:05.758066 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:05.759762 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:05.760516 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:05.762238 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:05.762667 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:05.762820 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:05.766674 (+ 3854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:05.766679 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:05.766683 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:05.766719 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:05.769957 (+ 3238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:05.770851 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:05.770855 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:05.770858 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:05.770978 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:05.771475 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:05.771479 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:05.771481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:05.771794 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:05.771957 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:05.772721 (+ 764us) server_negotiation.cc:299] Negotiation successful
0924 16:12:05.772957 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":335,"threads_started":1}
I20260924 16:12:05.774513 30882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:05.754630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:35839 (local address 127.29.234.1:58565)
0924 16:12:05.755632 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:05.755673 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:05.757524 (+ 1851us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:05.758338 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:05.758346 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:05.758952 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:05.759573 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:05.759593 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:05.760710 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:05.760715 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:05.762071 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:05.762084 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:05.762929 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:05.762938 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:05.763189 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:05.763937 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:05.763960 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:05.766401 (+ 2441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:05.770127 (+ 3726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:05.770138 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:05.770159 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:05.770703 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:05.771094 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:05.771099 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:05.771103 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:05.771320 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:05.771932 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:05.771941 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:05.772341 (+ 400us) client_negotiation.cc:769] Sending connection context
0924 16:12:05.772620 (+ 279us) client_negotiation.cc:240] Negotiation successful
0924 16:12:05.773227 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":712,"thread_start_us":419,"threads_started":1}
I20260924 16:12:05.776341 30879 heartbeater.cc:347] Connected to a master server at 127.29.234.62:35839
I20260924 16:12:05.776675 30879 heartbeater.cc:464] Registering TS with master...
I20260924 16:12:05.777529 30879 heartbeater.cc:511] Master 127.29.234.62:35839 requested a full tablet report, sending...
I20260924 16:12:05.779973 30685 ts_manager.cc:194] Registered new tserver with Master: 806828369087427eaa69c842f3503354 (127.29.234.1:38815)
I20260924 16:12:05.781674 30685 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.234.1@KRBTEST.COM'} at 127.29.234.1:58565
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab.
Entry for principal kudu/127.29.234.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab.
I20260924 16:12:05.794061 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.2:0
--local_ip_for_outbound_sockets=127.29.234.2
--webserver_interface=127.29.234.2
--webserver_port=0
--tserver_master_addrs=127.29.234.62:35839
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab
--principal=kudu/127.29.234.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:39439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:12:06.165668 30885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:06.165956 30885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:06.166054 30885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:06.173352 30885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:06.173528 30885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:06.173643 30885 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.2
I20260924 16:12:06.183790 30885 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:39439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab
--principal=kudu/127.29.234.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.234.2
--webserver_port=0
--tserver_master_addrs=127.29.234.62:35839
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.2
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:12:06.185037 30885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:06.186367 30885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:06.199534 30890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:06.199502 30892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:06.199644 30894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:06.200728 30885 server_base.cc:1034] running on GCE node
I20260924 16:12:06.201332 30885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:06.202299 30885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:06.203504 30885 hybrid_clock.cc:648] HybridClock initialized: now 1790266326203460 us; error 36 us; skew 500 ppm
I20260924 16:12:06.203835 30885 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:12:06 dist-test-slave-v5j9 krb5kdc[30640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266326, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:06.206992 30885 init.cc:377] Logged in from keytab as kudu/127.29.234.2@KRBTEST.COM (short username kudu)
I20260924 16:12:06.208477 30885 webserver.cc:460] Webserver started at http://127.29.234.2:37613/ using document root <none> and password file <none>
I20260924 16:12:06.209187 30885 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:06.209337 30885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:06.209704 30885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:06.212723 30885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/minicluster-data/ts-1/data/instance:
uuid: "de61b22926c5435b918daea2d16f1bb6"
format_stamp: "Formatted at 2026-09-24 16:12:06 on dist-test-slave-v5j9"
server_key: "84aaf3ecbfa9d60741a3f5f40c65a953"
server_key_iv: "29ab496c6d1958ca6a8c4fc22f44d778"
server_key_version: "encryptionkey@0"
I20260924 16:12:06.213567 30885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/minicluster-data/ts-1/wal/instance:
uuid: "de61b22926c5435b918daea2d16f1bb6"
format_stamp: "Formatted at 2026-09-24 16:12:06 on dist-test-slave-v5j9"
server_key: "84aaf3ecbfa9d60741a3f5f40c65a953"
server_key_iv: "29ab496c6d1958ca6a8c4fc22f44d778"
server_key_version: "encryptionkey@0"
I20260924 16:12:06.218871 30885 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260924 16:12:06.222556 30901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:06.223949 30885 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260924 16:12:06.224190 30885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/minicluster-data/ts-1/wal
uuid: "de61b22926c5435b918daea2d16f1bb6"
format_stamp: "Formatted at 2026-09-24 16:12:06 on dist-test-slave-v5j9"
server_key: "84aaf3ecbfa9d60741a3f5f40c65a953"
server_key_iv: "29ab496c6d1958ca6a8c4fc22f44d778"
server_key_version: "encryptionkey@0"
I20260924 16:12:06.224411 30885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:06.241417 30885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:06.245512 30885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:06.245797 30885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:06.246771 30885 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:12:06.248421 30885 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:12:06.248529 30885 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:06.248719 30885 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:12:06.248796 30885 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:06.286041 30885 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.2:43567
I20260924 16:12:06.286089 31015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.2:43567 every 8 connection(s)
I20260924 16:12:06.287391 30885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
I20260924 16:12:06.292299 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 30885
I20260924 16:12:06.292507 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/minicluster-data/ts-1/wal/instance
I20260924 16:12:06.293036 30632 external_mini_cluster.cc:1428] Setting key ae80d9c69583fc2d6b89dfde264f8379
Sep 24 16:12:06 dist-test-slave-v5j9 krb5kdc[30640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266326, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.2@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.234.3@KRBTEST.COM; defaulting to no policy
I20260924 16:12:06.310088 31018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:06.290034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.2:38165 (local address 127.29.234.62:35839)
0924 16:12:06.290481 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:06.290486 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:06.291413 (+ 927us) server_negotiation.cc:407] Connection header received
0924 16:12:06.294005 (+ 2592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:06.294011 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:06.294100 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:06.294220 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:06.296069 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:06.296803 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:06.299427 (+ 2624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:06.299871 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:06.300028 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:06.304150 (+ 4122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:06.304155 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:06.304158 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:06.304190 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:06.306959 (+ 2769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:06.307724 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:06.307728 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:06.307730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:06.307852 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:06.308291 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:06.308296 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:06.308299 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:06.308585 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:06.308740 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:06.309483 (+ 743us) server_negotiation.cc:299] Negotiation successful
0924 16:12:06.309706 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":242,"threads_started":1}
I20260924 16:12:06.311211 31019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:06.290502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:35839 (local address 127.29.234.2:38165)
0924 16:12:06.291240 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:06.291277 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:06.293673 (+ 2396us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:06.294489 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:06.294499 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:06.295093 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:06.295882 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:06.295910 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:06.296976 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:06.296984 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:06.299263 (+ 2279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:06.299281 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:06.300126 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:06.300133 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:06.300391 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:06.301122 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:06.301157 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:06.303904 (+ 2747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:06.307114 (+ 3210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:06.307124 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:06.307144 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:06.307589 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:06.307963 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:06.307968 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:06.307972 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:06.308171 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:06.308713 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:06.308722 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:06.309086 (+ 364us) client_negotiation.cc:769] Sending connection context
0924 16:12:06.309383 (+ 297us) client_negotiation.cc:240] Negotiation successful
0924 16:12:06.309963 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":295,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.3@KRBTEST.COM" created.
I20260924 16:12:06.312845 31016 heartbeater.cc:347] Connected to a master server at 127.29.234.62:35839
I20260924 16:12:06.313172 31016 heartbeater.cc:464] Registering TS with master...
I20260924 16:12:06.313978 31016 heartbeater.cc:511] Master 127.29.234.62:35839 requested a full tablet report, sending...
I20260924 16:12:06.315522 30685 ts_manager.cc:194] Registered new tserver with Master: de61b22926c5435b918daea2d16f1bb6 (127.29.234.2:43567)
I20260924 16:12:06.316696 30685 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.234.2@KRBTEST.COM'} at 127.29.234.2:38165
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab.
Entry for principal kudu/127.29.234.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab.
I20260924 16:12:06.334841 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.3:0
--local_ip_for_outbound_sockets=127.29.234.3
--webserver_interface=127.29.234.3
--webserver_port=0
--tserver_master_addrs=127.29.234.62:35839
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab
--principal=kudu/127.29.234.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:39439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:12:06.688832 31022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:06.689159 31022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:06.689334 31022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:06.696930 31022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:06.697119 31022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:06.697304 31022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.3
I20260924 16:12:06.707383 31022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:39439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab
--principal=kudu/127.29.234.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.234.3
--webserver_port=0
--tserver_master_addrs=127.29.234.62:35839
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.3
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:12:06.708752 31022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:06.710369 31022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:06.724450 31027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:06.724407 31028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:06.724366 31030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:06.725760 31022 server_base.cc:1034] running on GCE node
I20260924 16:12:06.726454 31022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:06.727418 31022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:06.728698 31022 hybrid_clock.cc:648] HybridClock initialized: now 1790266326728578 us; error 107 us; skew 500 ppm
I20260924 16:12:06.729063 31022 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:12:06 dist-test-slave-v5j9 krb5kdc[30640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266326, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:06.732259 31022 init.cc:377] Logged in from keytab as kudu/127.29.234.3@KRBTEST.COM (short username kudu)
I20260924 16:12:06.733893 31022 webserver.cc:460] Webserver started at http://127.29.234.3:44815/ using document root <none> and password file <none>
I20260924 16:12:06.734611 31022 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:06.734757 31022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:06.735080 31022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:06.738559 31022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/minicluster-data/ts-2/data/instance:
uuid: "47224ea0eb55463d898fe0b0b2fabf9d"
format_stamp: "Formatted at 2026-09-24 16:12:06 on dist-test-slave-v5j9"
server_key: "f4fdd36ceb62adfce91618459a5fff02"
server_key_iv: "c1bd05dead109524a7e0fa35b5f16a6f"
server_key_version: "encryptionkey@0"
I20260924 16:12:06.739601 31022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/minicluster-data/ts-2/wal/instance:
uuid: "47224ea0eb55463d898fe0b0b2fabf9d"
format_stamp: "Formatted at 2026-09-24 16:12:06 on dist-test-slave-v5j9"
server_key: "f4fdd36ceb62adfce91618459a5fff02"
server_key_iv: "c1bd05dead109524a7e0fa35b5f16a6f"
server_key_version: "encryptionkey@0"
I20260924 16:12:06.744933 31022 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 16:12:06.748397 31037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:06.750228 31022 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:12:06.750398 31022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/minicluster-data/ts-2/wal
uuid: "47224ea0eb55463d898fe0b0b2fabf9d"
format_stamp: "Formatted at 2026-09-24 16:12:06 on dist-test-slave-v5j9"
server_key: "f4fdd36ceb62adfce91618459a5fff02"
server_key_iv: "c1bd05dead109524a7e0fa35b5f16a6f"
server_key_version: "encryptionkey@0"
I20260924 16:12:06.750612 31022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:06.770521 31022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:06.774574 31022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:06.774811 31022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:06.775828 31022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:12:06.777105 31022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:12:06.777179 31022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:06.777266 31022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:12:06.777309 31022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:06.784873 30879 heartbeater.cc:503] Master 127.29.234.62:35839 was elected leader, sending a full tablet report...
I20260924 16:12:06.814644 31022 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.3:44135
I20260924 16:12:06.814692 31151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.3:44135 every 8 connection(s)
I20260924 16:12:06.815961 31022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
I20260924 16:12:06.823050 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 31022
I20260924 16:12:06.823256 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790266323169505-30632-0/minicluster-data/ts-2/wal/instance
I20260924 16:12:06.823817 30632 external_mini_cluster.cc:1428] Setting key ded7f946c14887d6c33c326fb075d528
Sep 24 16:12:06 dist-test-slave-v5j9 krb5kdc[30640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266326, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.3@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:12:06.837153 31154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:06.818664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.3:37549 (local address 127.29.234.62:35839)
0924 16:12:06.819133 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:06.819138 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:06.820201 (+ 1063us) server_negotiation.cc:407] Connection header received
0924 16:12:06.822154 (+ 1953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:06.822159 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:06.822248 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:06.822367 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:06.824511 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:06.825536 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:06.827755 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:06.828243 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:06.828338 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:06.831831 (+ 3493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:06.831836 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:06.831839 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:06.831873 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:06.834312 (+ 2439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:06.835114 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:06.835118 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:06.835121 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:06.835233 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:06.835608 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:06.835611 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:06.835613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:06.835872 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:06.835983 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:06.836708 (+ 725us) server_negotiation.cc:299] Negotiation successful
0924 16:12:06.836874 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":227,"threads_started":1}
I20260924 16:12:06.838182 31155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:06.819249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:35839 (local address 127.29.234.3:37549)
0924 16:12:06.820040 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:06.820087 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:06.821872 (+ 1785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:06.822634 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:06.822648 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:06.823462 (+ 814us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:06.824325 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:06.824351 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:06.825725 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:06.825730 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:06.827622 (+ 1892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:06.827634 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:06.828425 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:06.828430 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:06.828629 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:06.829450 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:06.829480 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:06.831641 (+ 2161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:06.834500 (+ 2859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:06.834507 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:06.834521 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:06.835009 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:06.835331 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:06.835335 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:06.835337 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:06.835515 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:06.835977 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:06.835983 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:06.836318 (+ 335us) client_negotiation.cc:769] Sending connection context
0924 16:12:06.836654 (+ 336us) client_negotiation.cc:240] Negotiation successful
0924 16:12:06.837053 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":262,"threads_started":1}
I20260924 16:12:06.839903 31152 heartbeater.cc:347] Connected to a master server at 127.29.234.62:35839
I20260924 16:12:06.840237 31152 heartbeater.cc:464] Registering TS with master...
I20260924 16:12:06.841080 31152 heartbeater.cc:511] Master 127.29.234.62:35839 requested a full tablet report, sending...
I20260924 16:12:06.842553 30685 ts_manager.cc:194] Registered new tserver with Master: 47224ea0eb55463d898fe0b0b2fabf9d (127.29.234.3:44135)
I20260924 16:12:06.843400 30685 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.234.3@KRBTEST.COM'} at 127.29.234.3:37549
I20260924 16:12:06.852893 30632 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 16:12:06 dist-test-slave-v5j9 krb5kdc[30640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266326, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 16:12:06 dist-test-slave-v5j9 krb5kdc[30640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266326, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.29.234.1@KRBTEST.COM
I20260924 16:12:06.899279 31160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:06.883545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51442 (local address 127.29.234.1:38815)
0924 16:12:06.884245 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:06.884256 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:06.884294 (+ 38us) server_negotiation.cc:407] Connection header received
0924 16:12:06.884382 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:06.884389 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:06.884617 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:06.884792 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:06.886397 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:06.887425 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:06.888717 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:06.889310 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:06.889504 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:06.893454 (+ 3950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:06.893466 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:06.893484 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:06.893530 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:06.896444 (+ 2914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:06.897160 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:06.897168 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:06.897170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:06.897299 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:06.897812 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:06.897816 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:06.897818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:06.898189 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:06.898431 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:06.898772 (+ 341us) server_negotiation.cc:299] Negotiation successful
0924 16:12:06.898991 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":328,"threads_started":1}
W20260924 16:12:06.900641 30793 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51442
Sep 24 16:12:06 dist-test-slave-v5j9 krb5kdc[30640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 16:12:06 dist-test-slave-v5j9 krb5kdc[30640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.234.1@KRBTEST.COM
I20260924 16:12:06.944988 31160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:06.930803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51454 (local address 127.29.234.1:38815)
0924 16:12:06.931052 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:06.931059 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:06.931218 (+ 159us) server_negotiation.cc:407] Connection header received
0924 16:12:06.931452 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:06.931456 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:06.931523 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:06.931628 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:06.932961 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:06.933745 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:06.934986 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:06.935434 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:06.935571 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:06.940314 (+ 4743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:06.940320 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:06.940323 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:06.940363 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:06.942899 (+ 2536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:06.943522 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:06.943526 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:06.943528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:06.943641 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:06.943938 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:06.943941 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:06.943942 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:06.944173 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:06.944342 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:06.944592 (+ 250us) server_negotiation.cc:299] Negotiation successful
0924 16:12:06.944750 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260924 16:12:06.946861 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 30747
I20260924 16:12:06.959285 31160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:06.947801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51466 (local address 127.29.234.1:38815)
0924 16:12:06.947949 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:06.947953 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:06.948304 (+ 351us) server_negotiation.cc:407] Connection header received
0924 16:12:06.948452 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:06.948455 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:06.948507 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:06.948575 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:06.949782 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:06.950491 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:06.951702 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:06.952096 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:06.952183 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:06.954415 (+ 2232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:06.954420 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:06.954424 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:06.954453 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:06.957055 (+ 2602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:06.957757 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:06.957761 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:06.957764 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:06.957887 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:06.958203 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:06.958206 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:06.958208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:06.958441 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:06.958613 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:06.958877 (+ 264us) server_negotiation.cc:299] Negotiation successful
0924 16:12:06.959027 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 16:12:06.960134 30874 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:12:07.044916 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 30747
I20260924 16:12:07.062319 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 30885
Sep 24 16:12:07 dist-test-slave-v5j9 krb5kdc[30640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.234.2@KRBTEST.COM
I20260924 16:12:07.078748 31167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:07.063267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42072 (local address 127.29.234.2:43567)
0924 16:12:07.063852 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:07.063859 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:07.063884 (+ 25us) server_negotiation.cc:407] Connection header received
0924 16:12:07.063954 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:07.063961 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:07.064170 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:07.064307 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:07.065568 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:07.066606 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:07.068064 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:07.068664 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:07.068761 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:07.072966 (+ 4205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:07.072979 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:07.072996 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:07.073032 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:07.076044 (+ 3012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:07.076665 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:07.076672 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:07.076675 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:07.076805 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:07.077155 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:07.077158 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:07.077160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:07.077608 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:07.077856 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:07.078099 (+ 243us) server_negotiation.cc:299] Negotiation successful
0924 16:12:07.078366 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":316,"threads_started":1}
I20260924 16:12:07.080036 31011 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:12:07.162048 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 30885
I20260924 16:12:07.178495 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 31022
Sep 24 16:12:07 dist-test-slave-v5j9 krb5kdc[30640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.234.3@KRBTEST.COM
I20260924 16:12:07.195106 31169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:07.179530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38138 (local address 127.29.234.3:44135)
0924 16:12:07.180146 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:07.180153 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:07.180181 (+ 28us) server_negotiation.cc:407] Connection header received
0924 16:12:07.180244 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:07.180252 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:07.180474 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:07.180654 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:07.182001 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:07.183027 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:07.184351 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:07.184923 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:07.185061 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:07.189263 (+ 4202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:07.189274 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:07.189290 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:07.189328 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:07.192094 (+ 2766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:07.192807 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:07.192824 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:07.192826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:07.192954 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:07.193317 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:07.193323 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:07.193327 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:07.193740 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:07.194085 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:07.194462 (+ 377us) server_negotiation.cc:299] Negotiation successful
0924 16:12:07.194747 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":310,"threads_started":1}
I20260924 16:12:07.196465 31147 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:12:07.281416 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 31022
I20260924 16:12:07.297726 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 30653
I20260924 16:12:07.299037 30716 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:12:07.406773 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 30653
2026-09-24T16:12:07Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4141 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:12:07 dist-test-slave-v5j9 krb5kdc[31174](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:12:07 dist-test-slave-v5j9 krb5kdc[31174](info): set up 2 sockets
Sep 24 16:12:07 dist-test-slave-v5j9 krb5kdc[31174](info): commencing operation
krb5kdc: starting...
W20260924 16:12:08.500571 30632 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Sep 24 16:12:08 dist-test-slave-v5j9 krb5kdc[31174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:12:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:12:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.234.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab.
Entry for principal kudu/127.29.234.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab.
I20260924 16:12:08.688637 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:44465
--webserver_interface=127.29.234.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab
--principal=kudu/127.29.234.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:42459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.234.62:44465
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:12:09.012661 31187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:09.013043 31187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:09.013211 31187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:09.020624 31187 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:12:09.020789 31187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:09.020902 31187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:09.021003 31187 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:12:09.021085 31187 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:12:09.031306 31187 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:42459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.234.62:44465
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab
--principal=kudu/127.29.234.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:44465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.234.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:12:09.032709 31187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:09.034150 31187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:09.046096 31193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:09.046229 31192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:09.046159 31195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:09.047098 31187 server_base.cc:1034] running on GCE node
I20260924 16:12:09.047957 31187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:09.049458 31187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:09.050699 31187 hybrid_clock.cc:648] HybridClock initialized: now 1790266329050644 us; error 43 us; skew 500 ppm
I20260924 16:12:09.051081 31187 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:12:09 dist-test-slave-v5j9 krb5kdc[31174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266329, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:09.054481 31187 init.cc:377] Logged in from keytab as kudu/127.29.234.62@KRBTEST.COM (short username kudu)
I20260924 16:12:09.056277 31187 webserver.cc:460] Webserver started at http://127.29.234.62:43677/ using document root <none> and password file <none>
I20260924 16:12:09.057051 31187 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:09.057206 31187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:09.057621 31187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:09.060762 31187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/minicluster-data/master-0/data/instance:
uuid: "bd52b4094e744f839b7a4b16f056cd60"
format_stamp: "Formatted at 2026-09-24 16:12:09 on dist-test-slave-v5j9"
server_key: "dc377ede04d00ae4e3f09c98848b748d"
server_key_iv: "a535c981cc31528ac0859a9439206bb1"
server_key_version: "encryptionkey@0"
I20260924 16:12:09.061686 31187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/minicluster-data/master-0/wal/instance:
uuid: "bd52b4094e744f839b7a4b16f056cd60"
format_stamp: "Formatted at 2026-09-24 16:12:09 on dist-test-slave-v5j9"
server_key: "dc377ede04d00ae4e3f09c98848b748d"
server_key_iv: "a535c981cc31528ac0859a9439206bb1"
server_key_version: "encryptionkey@0"
I20260924 16:12:09.067114 31187 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 16:12:09.070721 31202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:09.072134 31187 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:12:09.072332 31187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/minicluster-data/master-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/minicluster-data/master-0/wal
uuid: "bd52b4094e744f839b7a4b16f056cd60"
format_stamp: "Formatted at 2026-09-24 16:12:09 on dist-test-slave-v5j9"
server_key: "dc377ede04d00ae4e3f09c98848b748d"
server_key_iv: "a535c981cc31528ac0859a9439206bb1"
server_key_version: "encryptionkey@0"
I20260924 16:12:09.072551 31187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:09.087985 31187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:09.092408 31187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:09.092720 31187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:09.111954 31187 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.62:44465
I20260924 16:12:09.111961 31254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.62:44465 every 8 connection(s)
I20260924 16:12:09.113490 31187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
I20260924 16:12:09.115742 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 31187
I20260924 16:12:09.115973 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/minicluster-data/master-0/wal/instance
I20260924 16:12:09.116621 30632 external_mini_cluster.cc:1428] Setting key f61d54f42efa20cec9dab6b2aea15ea7
I20260924 16:12:09.117746 31255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 16:12:09 dist-test-slave-v5j9 krb5kdc[31174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:12:09.132187 31255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd52b4094e744f839b7a4b16f056cd60: Bootstrap starting.
I20260924 16:12:09.136519 31255 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd52b4094e744f839b7a4b16f056cd60: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:09.137887 31255 log.cc:824] T 00000000000000000000000000000000 P bd52b4094e744f839b7a4b16f056cd60: Log is configured to *not* fsync() on all Append() calls
I20260924 16:12:09.141253 31255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd52b4094e744f839b7a4b16f056cd60: No bootstrap required, opened a new log
I20260924 16:12:09.141602 31258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:09.118858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34196 (local address 127.29.234.62:44465)
0924 16:12:09.119717 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:09.119735 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:09.119789 (+ 54us) server_negotiation.cc:407] Connection header received
0924 16:12:09.121632 (+ 1843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:09.121662 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:09.122222 (+ 560us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:09.122709 (+ 487us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:09.124113 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:09.126219 (+ 2106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:09.127440 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:09.128476 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:09.128677 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:09.132855 (+ 4178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:09.132865 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:09.132886 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:09.132930 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:09.136953 (+ 4023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:09.137703 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:09.137713 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:09.137722 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:09.137944 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:09.138317 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:09.138325 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:09.138330 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:09.138989 (+ 659us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:09.139288 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:09.139714 (+ 426us) server_negotiation.cc:299] Negotiation successful
0924 16:12:09.140146 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":363,"threads_started":1}
I20260924 16:12:09.149931 31255 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd52b4094e744f839b7a4b16f056cd60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd52b4094e744f839b7a4b16f056cd60" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 44465 } }
I20260924 16:12:09.150311 31255 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd52b4094e744f839b7a4b16f056cd60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:09.150388 31255 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd52b4094e744f839b7a4b16f056cd60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd52b4094e744f839b7a4b16f056cd60, State: Initialized, Role: FOLLOWER
I20260924 16:12:09.150997 31255 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd52b4094e744f839b7a4b16f056cd60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd52b4094e744f839b7a4b16f056cd60" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 44465 } }
I20260924 16:12:09.151206 31255 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd52b4094e744f839b7a4b16f056cd60 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:12:09.151304 31255 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd52b4094e744f839b7a4b16f056cd60 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:12:09.151463 31255 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd52b4094e744f839b7a4b16f056cd60 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:09.154120 31255 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd52b4094e744f839b7a4b16f056cd60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd52b4094e744f839b7a4b16f056cd60" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 44465 } }
I20260924 16:12:09.154644 31255 leader_election.cc:302] T 00000000000000000000000000000000 P bd52b4094e744f839b7a4b16f056cd60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd52b4094e744f839b7a4b16f056cd60; no voters:
I20260924 16:12:09.155259 31255 leader_election.cc:288] T 00000000000000000000000000000000 P bd52b4094e744f839b7a4b16f056cd60 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:12:09.155411 31260 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd52b4094e744f839b7a4b16f056cd60 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:12:09.156141 31260 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd52b4094e744f839b7a4b16f056cd60 [term 1 LEADER]: Becoming Leader. State: Replica: bd52b4094e744f839b7a4b16f056cd60, State: Running, Role: LEADER
I20260924 16:12:09.156737 31260 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd52b4094e744f839b7a4b16f056cd60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd52b4094e744f839b7a4b16f056cd60" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 44465 } }
I20260924 16:12:09.157500 31255 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd52b4094e744f839b7a4b16f056cd60 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:12:09.159801 31261 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd52b4094e744f839b7a4b16f056cd60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd52b4094e744f839b7a4b16f056cd60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd52b4094e744f839b7a4b16f056cd60" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 44465 } } }
I20260924 16:12:09.160245 31261 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd52b4094e744f839b7a4b16f056cd60 [sys.catalog]: This master's current role is: LEADER
I20260924 16:12:09.160770 31262 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd52b4094e744f839b7a4b16f056cd60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd52b4094e744f839b7a4b16f056cd60. Latest consensus state: current_term: 1 leader_uuid: "bd52b4094e744f839b7a4b16f056cd60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd52b4094e744f839b7a4b16f056cd60" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 44465 } } }
I20260924 16:12:09.161165 31262 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd52b4094e744f839b7a4b16f056cd60 [sys.catalog]: This master's current role is: LEADER
I20260924 16:12:09.162953 31267 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:12:09.168414 31267 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:12:09.178702 31267 catalog_manager.cc:1347] Generated new cluster ID: 8984813f0e10455fafdcb962af674f22
I20260924 16:12:09.178861 31267 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:12:09.190158 31267 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:12:09.191522 31267 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:12:09.204821 31267 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd52b4094e744f839b7a4b16f056cd60: Generated new TSK 0
I20260924 16:12:09.205792 31267 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.234.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab.
Entry for principal kudu/127.29.234.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab.
I20260924 16:12:09.255419 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.1:0
--local_ip_for_outbound_sockets=127.29.234.1
--webserver_interface=127.29.234.1
--webserver_port=0
--tserver_master_addrs=127.29.234.62:44465
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab
--principal=kudu/127.29.234.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:42459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:12:09.610682 31282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:09.610966 31282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:09.611068 31282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:09.618486 31282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:09.618609 31282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:09.618736 31282 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.1
I20260924 16:12:09.628787 31282 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:42459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab
--principal=kudu/127.29.234.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.234.1
--webserver_port=0
--tserver_master_addrs=127.29.234.62:44465
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.1
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:12:09.630049 31282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:09.631377 31282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:09.644917 31287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:09.644942 31288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:09.645134 31290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:09.645879 31282 server_base.cc:1034] running on GCE node
I20260924 16:12:09.646577 31282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:09.647529 31282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:09.648742 31282 hybrid_clock.cc:648] HybridClock initialized: now 1790266329648692 us; error 37 us; skew 500 ppm
I20260924 16:12:09.649050 31282 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:12:09 dist-test-slave-v5j9 krb5kdc[31174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266329, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:09.652305 31282 init.cc:377] Logged in from keytab as kudu/127.29.234.1@KRBTEST.COM (short username kudu)
I20260924 16:12:09.653844 31282 webserver.cc:460] Webserver started at http://127.29.234.1:34407/ using document root <none> and password file <none>
I20260924 16:12:09.654675 31282 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:09.654832 31282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:09.655149 31282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:09.658373 31282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/minicluster-data/ts-0/data/instance:
uuid: "37c39763e45a47c79a38a7be18974bc9"
format_stamp: "Formatted at 2026-09-24 16:12:09 on dist-test-slave-v5j9"
server_key: "7a8ce97c54ca349694f1b9a4e4217e8b"
server_key_iv: "389c3408e6d15f85fd6261e3d11e98ff"
server_key_version: "encryptionkey@0"
I20260924 16:12:09.659332 31282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/minicluster-data/ts-0/wal/instance:
uuid: "37c39763e45a47c79a38a7be18974bc9"
format_stamp: "Formatted at 2026-09-24 16:12:09 on dist-test-slave-v5j9"
server_key: "7a8ce97c54ca349694f1b9a4e4217e8b"
server_key_iv: "389c3408e6d15f85fd6261e3d11e98ff"
server_key_version: "encryptionkey@0"
I20260924 16:12:09.664721 31282 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 16:12:09.668949 31297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:09.670588 31282 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 16:12:09.670759 31282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/minicluster-data/ts-0/wal
uuid: "37c39763e45a47c79a38a7be18974bc9"
format_stamp: "Formatted at 2026-09-24 16:12:09 on dist-test-slave-v5j9"
server_key: "7a8ce97c54ca349694f1b9a4e4217e8b"
server_key_iv: "389c3408e6d15f85fd6261e3d11e98ff"
server_key_version: "encryptionkey@0"
I20260924 16:12:09.670965 31282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:09.689509 31282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:09.693756 31282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:09.694005 31282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:09.694968 31282 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:12:09.696403 31282 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:12:09.696475 31282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:09.696563 31282 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:12:09.696606 31282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:09.730039 31282 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.1:45941
I20260924 16:12:09.730075 31410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.1:45941 every 8 connection(s)
I20260924 16:12:09.731395 31282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
I20260924 16:12:09.736526 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 31282
I20260924 16:12:09.736774 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/minicluster-data/ts-0/wal/instance
I20260924 16:12:09.737504 30632 external_mini_cluster.cc:1428] Setting key 50a6c3567ee01ebcbedb938ece0b54a1
Sep 24 16:12:09 dist-test-slave-v5j9 krb5kdc[31174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266329, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.1@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:12:09.755162 31413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:09.734088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.1:34057 (local address 127.29.234.62:44465)
0924 16:12:09.734584 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:09.734589 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:09.735512 (+ 923us) server_negotiation.cc:407] Connection header received
0924 16:12:09.737454 (+ 1942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:09.737459 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:09.737554 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:09.737676 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:09.739773 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:09.740487 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:09.743476 (+ 2989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:09.743882 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:09.743990 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:09.748096 (+ 4106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:09.748123 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:09.748132 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:09.748177 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:09.751764 (+ 3587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:09.752527 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:09.752536 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:09.752542 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:09.752695 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:09.753143 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:09.753150 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:09.753157 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:09.753618 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:09.753793 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:09.754504 (+ 711us) server_negotiation.cc:299] Negotiation successful
0924 16:12:09.754756 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":279,"threads_started":1}
I20260924 16:12:09.756078 31414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:09.734625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:44465 (local address 127.29.234.1:34057)
0924 16:12:09.735344 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:09.735387 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:09.737159 (+ 1772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:09.738024 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:09.738036 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:09.738721 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:09.739555 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:09.739578 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:09.740680 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:09.740689 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:09.743305 (+ 2616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:09.743324 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:09.744118 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:09.744127 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:09.744476 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:09.745332 (+ 856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:09.745414 (+ 82us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:09.747867 (+ 2453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:09.751924 (+ 4057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:09.751934 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:09.751953 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:09.752393 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:09.752812 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:09.752815 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:09.752817 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:09.752992 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:09.753760 (+ 768us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:09.753767 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:09.754071 (+ 304us) client_negotiation.cc:769] Sending connection context
0924 16:12:09.754326 (+ 255us) client_negotiation.cc:240] Negotiation successful
0924 16:12:09.754798 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":274,"threads_started":1}
WARNING: no policy specified for kudu/127.29.234.2@KRBTEST.COM; defaulting to no policy
I20260924 16:12:09.757781 31411 heartbeater.cc:347] Connected to a master server at 127.29.234.62:44465
I20260924 16:12:09.758121 31411 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.2@KRBTEST.COM" created.
I20260924 16:12:09.759047 31411 heartbeater.cc:511] Master 127.29.234.62:44465 requested a full tablet report, sending...
I20260924 16:12:09.761255 31220 ts_manager.cc:194] Registered new tserver with Master: 37c39763e45a47c79a38a7be18974bc9 (127.29.234.1:45941)
I20260924 16:12:09.762741 31220 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.234.1@KRBTEST.COM'} at 127.29.234.1:34057
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab.
Entry for principal kudu/127.29.234.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab.
I20260924 16:12:09.780584 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.2:0
--local_ip_for_outbound_sockets=127.29.234.2
--webserver_interface=127.29.234.2
--webserver_port=0
--tserver_master_addrs=127.29.234.62:44465
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab
--principal=kudu/127.29.234.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:42459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:12:10.111640 31417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:10.111965 31417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:10.112134 31417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:10.119493 31417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:10.119664 31417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:10.119860 31417 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.2
I20260924 16:12:10.129573 31417 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:42459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab
--principal=kudu/127.29.234.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.234.2
--webserver_port=0
--tserver_master_addrs=127.29.234.62:44465
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.2
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:12:10.130925 31417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:10.132266 31417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:10.145468 31422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:10.145493 31425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:10.145530 31423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:10.146435 31417 server_base.cc:1034] running on GCE node
I20260924 16:12:10.147013 31417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:10.147965 31417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:10.149209 31417 hybrid_clock.cc:648] HybridClock initialized: now 1790266330149159 us; error 38 us; skew 500 ppm
I20260924 16:12:10.149578 31417 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:12:10 dist-test-slave-v5j9 krb5kdc[31174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266330, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:10.160487 31417 init.cc:377] Logged in from keytab as kudu/127.29.234.2@KRBTEST.COM (short username kudu)
I20260924 16:12:10.162246 31417 webserver.cc:460] Webserver started at http://127.29.234.2:38443/ using document root <none> and password file <none>
I20260924 16:12:10.163091 31417 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:10.163267 31417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:10.163600 31417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:10.167007 31417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/minicluster-data/ts-1/data/instance:
uuid: "1b954844353e4df7953691215344770f"
format_stamp: "Formatted at 2026-09-24 16:12:10 on dist-test-slave-v5j9"
server_key: "73fafd41dab854bc0e7ea2126b97a62b"
server_key_iv: "708023de17081782e60dc471e7be6da9"
server_key_version: "encryptionkey@0"
I20260924 16:12:10.167842 31417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/minicluster-data/ts-1/wal/instance:
uuid: "1b954844353e4df7953691215344770f"
format_stamp: "Formatted at 2026-09-24 16:12:10 on dist-test-slave-v5j9"
server_key: "73fafd41dab854bc0e7ea2126b97a62b"
server_key_iv: "708023de17081782e60dc471e7be6da9"
server_key_version: "encryptionkey@0"
I20260924 16:12:10.173759 31417 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260924 16:12:10.177575 31434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:10.178939 31417 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:12:10.179147 31417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/minicluster-data/ts-1/wal
uuid: "1b954844353e4df7953691215344770f"
format_stamp: "Formatted at 2026-09-24 16:12:10 on dist-test-slave-v5j9"
server_key: "73fafd41dab854bc0e7ea2126b97a62b"
server_key_iv: "708023de17081782e60dc471e7be6da9"
server_key_version: "encryptionkey@0"
I20260924 16:12:10.179373 31417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:10.206462 31417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:10.213332 31417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:10.213685 31417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:10.214715 31417 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:12:10.216339 31417 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:12:10.216444 31417 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:10.216575 31417 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:12:10.216641 31417 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:10.253772 31417 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.2:38325
I20260924 16:12:10.253840 31547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.2:38325 every 8 connection(s)
I20260924 16:12:10.255337 31417 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
I20260924 16:12:10.258428 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 31417
I20260924 16:12:10.258623 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/minicluster-data/ts-1/wal/instance
I20260924 16:12:10.259132 30632 external_mini_cluster.cc:1428] Setting key 59d0d76bf0927e962454883841bd8c01
Sep 24 16:12:10 dist-test-slave-v5j9 krb5kdc[31174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266330, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.2@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.234.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.3@KRBTEST.COM" created.
I20260924 16:12:10.286630 31551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:10.262064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.2:48935 (local address 127.29.234.62:44465)
0924 16:12:10.262495 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:10.262505 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:10.262525 (+ 20us) server_negotiation.cc:407] Connection header received
0924 16:12:10.268150 (+ 5625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:10.268154 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:10.268222 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:10.268343 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:10.270513 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.271264 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.273847 (+ 2583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.274336 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.274426 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:10.279390 (+ 4964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:10.279400 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:10.279409 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:10.279457 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:10.283146 (+ 3689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:10.283944 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:10.283952 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:10.283959 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:10.284116 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:10.284531 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:10.284539 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:10.284545 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:10.284956 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:10.285117 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:10.285861 (+ 744us) server_negotiation.cc:299] Negotiation successful
0924 16:12:10.286115 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":1190,"threads_started":1}
I20260924 16:12:10.287508 31550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:10.261012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:44465 (local address 127.29.234.2:48935)
0924 16:12:10.261842 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:10.261894 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:10.264175 (+ 2281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:10.268620 (+ 4445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:10.268629 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:10.269481 (+ 852us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:10.270303 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:10.270335 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.271423 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.271432 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:10.273687 (+ 2255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:10.273705 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.274558 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.274568 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:10.274892 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:10.276405 (+ 1513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:10.276428 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:10.279112 (+ 2684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:10.283298 (+ 4186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:10.283305 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:10.283319 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:10.283811 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:10.284235 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:10.284238 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:10.284240 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:10.284414 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:10.285082 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:10.285088 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:10.285485 (+ 397us) client_negotiation.cc:769] Sending connection context
0924 16:12:10.285712 (+ 227us) client_negotiation.cc:240] Negotiation successful
0924 16:12:10.286095 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":1273,"threads_started":1}
I20260924 16:12:10.289096 31548 heartbeater.cc:347] Connected to a master server at 127.29.234.62:44465
I20260924 16:12:10.289472 31548 heartbeater.cc:464] Registering TS with master...
I20260924 16:12:10.290194 31548 heartbeater.cc:511] Master 127.29.234.62:44465 requested a full tablet report, sending...
I20260924 16:12:10.291674 31220 ts_manager.cc:194] Registered new tserver with Master: 1b954844353e4df7953691215344770f (127.29.234.2:38325)
I20260924 16:12:10.292999 31220 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.234.2@KRBTEST.COM'} at 127.29.234.2:48935
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab.
Entry for principal kudu/127.29.234.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab.
I20260924 16:12:10.306068 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.3:0
--local_ip_for_outbound_sockets=127.29.234.3
--webserver_interface=127.29.234.3
--webserver_port=0
--tserver_master_addrs=127.29.234.62:44465
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab
--principal=kudu/127.29.234.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:42459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:12:10.642673 31554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:10.642999 31554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:10.643179 31554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:10.650501 31554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:10.650684 31554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:10.650883 31554 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.3
I20260924 16:12:10.661336 31554 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:42459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab
--principal=kudu/127.29.234.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.234.3
--webserver_port=0
--tserver_master_addrs=127.29.234.62:44465
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.3
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:12:10.662775 31554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:10.664166 31554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:10.677767 31560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:10.677946 31559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:10.677739 31562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:10.678936 31554 server_base.cc:1034] running on GCE node
I20260924 16:12:10.679553 31554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:10.680482 31554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:10.681715 31554 hybrid_clock.cc:648] HybridClock initialized: now 1790266330681660 us; error 43 us; skew 500 ppm
I20260924 16:12:10.682034 31554 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:12:10 dist-test-slave-v5j9 krb5kdc[31174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266330, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:10.685330 31554 init.cc:377] Logged in from keytab as kudu/127.29.234.3@KRBTEST.COM (short username kudu)
I20260924 16:12:10.686808 31554 webserver.cc:460] Webserver started at http://127.29.234.3:39043/ using document root <none> and password file <none>
I20260924 16:12:10.687515 31554 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:10.687614 31554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:10.687969 31554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:10.690971 31554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/minicluster-data/ts-2/data/instance:
uuid: "a4eeb08a503b4ce4a1bd4d3696a15668"
format_stamp: "Formatted at 2026-09-24 16:12:10 on dist-test-slave-v5j9"
server_key: "c6cd222e6c4864e335e2b2828c99f2bf"
server_key_iv: "0bb64021b0ae08eee0f7ecc65c8460fe"
server_key_version: "encryptionkey@0"
I20260924 16:12:10.691762 31554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/minicluster-data/ts-2/wal/instance:
uuid: "a4eeb08a503b4ce4a1bd4d3696a15668"
format_stamp: "Formatted at 2026-09-24 16:12:10 on dist-test-slave-v5j9"
server_key: "c6cd222e6c4864e335e2b2828c99f2bf"
server_key_iv: "0bb64021b0ae08eee0f7ecc65c8460fe"
server_key_version: "encryptionkey@0"
I20260924 16:12:10.696871 31554 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260924 16:12:10.700410 31569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:10.701903 31554 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 16:12:10.702075 31554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/minicluster-data/ts-2/wal
uuid: "a4eeb08a503b4ce4a1bd4d3696a15668"
format_stamp: "Formatted at 2026-09-24 16:12:10 on dist-test-slave-v5j9"
server_key: "c6cd222e6c4864e335e2b2828c99f2bf"
server_key_iv: "0bb64021b0ae08eee0f7ecc65c8460fe"
server_key_version: "encryptionkey@0"
I20260924 16:12:10.702253 31554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:10.720396 31554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:10.725144 31554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:10.725481 31554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:10.726538 31554 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:12:10.728005 31554 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:12:10.728077 31554 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:10.728166 31554 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:12:10.728209 31554 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:10.762959 31554 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.3:34251
I20260924 16:12:10.762977 31683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.3:34251 every 8 connection(s)
I20260924 16:12:10.764273 31554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
I20260924 16:12:10.765630 31411 heartbeater.cc:503] Master 127.29.234.62:44465 was elected leader, sending a full tablet report...
I20260924 16:12:10.774101 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 31554
I20260924 16:12:10.774341 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790266323169505-30632-0/minicluster-data/ts-2/wal/instance
I20260924 16:12:10.774896 30632 external_mini_cluster.cc:1428] Setting key ece7080446624ec91fc898a8a6b3d895
Sep 24 16:12:10 dist-test-slave-v5j9 krb5kdc[31174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266330, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.3@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:12:10.784538 31551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:10.767128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.3:55055 (local address 127.29.234.62:44465)
0924 16:12:10.767317 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:10.767326 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:10.768622 (+ 1296us) server_negotiation.cc:407] Connection header received
0924 16:12:10.770433 (+ 1811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:10.770437 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:10.770508 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:10.770593 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:10.772077 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.772818 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.774440 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.775111 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.775245 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:10.779303 (+ 4058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:10.779307 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:10.779310 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:10.779342 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:10.781834 (+ 2492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:10.782489 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:10.782492 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:10.782494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:10.782603 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:10.782991 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:10.782994 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:10.782996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:10.783268 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:10.783390 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:10.784151 (+ 761us) server_negotiation.cc:299] Negotiation successful
0924 16:12:10.784317 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 16:12:10.785578 31686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:10.767771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:44465 (local address 127.29.234.3:55055)
0924 16:12:10.768465 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:10.768511 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:10.770182 (+ 1671us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:10.770802 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:10.770810 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:10.771355 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:10.771931 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:10.771949 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.772941 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.772944 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:10.774291 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:10.774308 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.775362 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.775370 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:10.775683 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:10.776576 (+ 893us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:10.776619 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:10.779114 (+ 2495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:10.781954 (+ 2840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:10.781960 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:10.781974 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:10.782385 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:10.782689 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:10.782692 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:10.782694 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:10.782864 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:10.783396 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:10.783402 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:10.783766 (+ 364us) client_negotiation.cc:769] Sending connection context
0924 16:12:10.783999 (+ 233us) client_negotiation.cc:240] Negotiation successful
0924 16:12:10.784391 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":260,"threads_started":1}
I20260924 16:12:10.787071 31684 heartbeater.cc:347] Connected to a master server at 127.29.234.62:44465
I20260924 16:12:10.787389 31684 heartbeater.cc:464] Registering TS with master...
I20260924 16:12:10.788170 31684 heartbeater.cc:511] Master 127.29.234.62:44465 requested a full tablet report, sending...
I20260924 16:12:10.789872 31220 ts_manager.cc:194] Registered new tserver with Master: a4eeb08a503b4ce4a1bd4d3696a15668 (127.29.234.3:34251)
I20260924 16:12:10.790757 30632 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:12:10.790784 31220 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.234.3@KRBTEST.COM'} at 127.29.234.3:55055
I20260924 16:12:10.810441 31551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:10.797020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34204 (local address 127.29.234.62:44465)
0924 16:12:10.797167 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:10.797172 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:10.797731 (+ 559us) server_negotiation.cc:407] Connection header received
0924 16:12:10.797867 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:10.797871 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:10.797932 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:10.798011 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:10.800267 (+ 2256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.801266 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.802570 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.802963 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.803055 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:10.805330 (+ 2275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:10.805335 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:10.805338 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:10.805457 (+ 119us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:10.808112 (+ 2655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:10.808708 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:10.808711 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:10.808714 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:10.808823 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:10.809120 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:10.809123 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:10.809125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:10.809533 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:10.809715 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:10.810011 (+ 296us) server_negotiation.cc:299] Negotiation successful
0924 16:12:10.810166 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 16:12:10.817181 31220 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34204:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:12:10.824095 31220 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:12:10.863742 31698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:10.837606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58626 (local address 127.29.234.2:38325)
0924 16:12:10.852630 (+ 15024us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:10.852640 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:10.853501 (+ 861us) server_negotiation.cc:407] Connection header received
0924 16:12:10.853618 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:10.853627 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:10.853957 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:10.854201 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:12:10.855151 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.857403 (+ 2252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.860726 (+ 3323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.862326 (+ 1600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.862466 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:10.862686 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:10.863227 (+ 541us) server_negotiation.cc:299] Negotiation successful
0924 16:12:10.863400 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14838,"thread_start_us":14712,"threads_started":1}
I20260924 16:12:10.863798 31697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:10.836867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.2:38325 (local address 127.0.0.1:58626)
0924 16:12:10.852754 (+ 15887us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:10.852813 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:10.853026 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:10.854503 (+ 1477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:10.854511 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:10.854552 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:12:10.854965 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:10.854979 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.857602 (+ 2623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.857611 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:10.860457 (+ 2846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:10.860477 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.862576 (+ 2099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.862584 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:10.862921 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:10.863003 (+ 82us) client_negotiation.cc:769] Sending connection context
0924 16:12:10.863146 (+ 143us) client_negotiation.cc:240] Negotiation successful
0924 16:12:10.863339 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15581,"mutex_wait_us":33,"thread_start_us":15899,"threads_started":1}
I20260924 16:12:10.866888 31699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:10.837600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53234 (local address 127.29.234.3:34251)
0924 16:12:10.858939 (+ 21339us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:10.858948 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:10.858982 (+ 34us) server_negotiation.cc:407] Connection header received
0924 16:12:10.859082 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:10.859089 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:10.859330 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:10.859530 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:12:10.860237 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.861907 (+ 1670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.864420 (+ 2513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.865775 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.865919 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:10.866180 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:10.866406 (+ 226us) server_negotiation.cc:299] Negotiation successful
0924 16:12:10.866565 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21157,"thread_start_us":20721,"threads_started":1}
I20260924 16:12:10.866928 31696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:10.837066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.3:34251 (local address 127.0.0.1:53234)
0924 16:12:10.853162 (+ 16096us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:10.853187 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:10.853388 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:10.859535 (+ 6147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:10.859539 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:10.859560 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:12:10.859973 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:10.859981 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.862039 (+ 2058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.862045 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:10.864233 (+ 2188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:10.864254 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.865984 (+ 1730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.865990 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:10.866182 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:10.866227 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 16:12:10.866360 (+ 133us) client_negotiation.cc:240] Negotiation successful
0924 16:12:10.866484 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15920,"mutex_wait_us":3,"thread_start_us":16105,"threads_started":1}
I20260924 16:12:10.868589 31482 tablet_service.cc:1467] Processing CreateTablet for tablet 2172a7a615d441d1ae2b6dedcb08dfbf (DEFAULT_TABLE table=test-table [id=f7e87c535c05449eb9f0a9be7bdb9aae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:12:10.869381 31700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:10.853694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:45941 (local address 127.0.0.1:43708)
0924 16:12:10.861534 (+ 7840us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:10.861551 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:10.861666 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:10.862223 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:10.862229 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:10.862265 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:12:10.862690 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:10.862703 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.864586 (+ 1883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.864591 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:10.866359 (+ 1768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:10.866373 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.868628 (+ 2255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.868635 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:10.868800 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:10.868830 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 16:12:10.868938 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 16:12:10.869053 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7728,"thread_start_us":7538,"threads_started":1}
I20260924 16:12:10.869817 31701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:10.854178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43708 (local address 127.29.234.1:45941)
0924 16:12:10.859097 (+ 4919us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:10.859104 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:10.861670 (+ 2566us) server_negotiation.cc:407] Connection header received
0924 16:12:10.861783 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:10.861791 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:10.862023 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:10.862220 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:12:10.862891 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.864414 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.866549 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:10.868142 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:10.868272 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:10.868489 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:10.869021 (+ 532us) server_negotiation.cc:299] Negotiation successful
0924 16:12:10.869204 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4758,"thread_start_us":4583,"threads_started":1}
I20260924 16:12:10.870173 31482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2172a7a615d441d1ae2b6dedcb08dfbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:10.870894 31618 tablet_service.cc:1467] Processing CreateTablet for tablet 2172a7a615d441d1ae2b6dedcb08dfbf (DEFAULT_TABLE table=test-table [id=f7e87c535c05449eb9f0a9be7bdb9aae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:12:10.872228 31618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2172a7a615d441d1ae2b6dedcb08dfbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:10.873237 31345 tablet_service.cc:1467] Processing CreateTablet for tablet 2172a7a615d441d1ae2b6dedcb08dfbf (DEFAULT_TABLE table=test-table [id=f7e87c535c05449eb9f0a9be7bdb9aae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:12:10.874558 31345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2172a7a615d441d1ae2b6dedcb08dfbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:10.884058 31702 tablet_bootstrap.cc:492] T 2172a7a615d441d1ae2b6dedcb08dfbf P a4eeb08a503b4ce4a1bd4d3696a15668: Bootstrap starting.
I20260924 16:12:10.886873 31703 tablet_bootstrap.cc:492] T 2172a7a615d441d1ae2b6dedcb08dfbf P 1b954844353e4df7953691215344770f: Bootstrap starting.
I20260924 16:12:10.887786 31702 tablet_bootstrap.cc:654] T 2172a7a615d441d1ae2b6dedcb08dfbf P a4eeb08a503b4ce4a1bd4d3696a15668: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:10.888823 31702 log.cc:824] T 2172a7a615d441d1ae2b6dedcb08dfbf P a4eeb08a503b4ce4a1bd4d3696a15668: Log is configured to *not* fsync() on all Append() calls
I20260924 16:12:10.889746 31704 tablet_bootstrap.cc:492] T 2172a7a615d441d1ae2b6dedcb08dfbf P 37c39763e45a47c79a38a7be18974bc9: Bootstrap starting.
I20260924 16:12:10.890476 31703 tablet_bootstrap.cc:654] T 2172a7a615d441d1ae2b6dedcb08dfbf P 1b954844353e4df7953691215344770f: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:10.891739 31703 log.cc:824] T 2172a7a615d441d1ae2b6dedcb08dfbf P 1b954844353e4df7953691215344770f: Log is configured to *not* fsync() on all Append() calls
I20260924 16:12:10.893209 31702 tablet_bootstrap.cc:492] T 2172a7a615d441d1ae2b6dedcb08dfbf P a4eeb08a503b4ce4a1bd4d3696a15668: No bootstrap required, opened a new log
I20260924 16:12:10.893515 31702 ts_tablet_manager.cc:1397] T 2172a7a615d441d1ae2b6dedcb08dfbf P a4eeb08a503b4ce4a1bd4d3696a15668: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20260924 16:12:10.893623 31704 tablet_bootstrap.cc:654] T 2172a7a615d441d1ae2b6dedcb08dfbf P 37c39763e45a47c79a38a7be18974bc9: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:10.894820 31704 log.cc:824] T 2172a7a615d441d1ae2b6dedcb08dfbf P 37c39763e45a47c79a38a7be18974bc9: Log is configured to *not* fsync() on all Append() calls
I20260924 16:12:10.898054 31704 tablet_bootstrap.cc:492] T 2172a7a615d441d1ae2b6dedcb08dfbf P 37c39763e45a47c79a38a7be18974bc9: No bootstrap required, opened a new log
I20260924 16:12:10.898053 31703 tablet_bootstrap.cc:492] T 2172a7a615d441d1ae2b6dedcb08dfbf P 1b954844353e4df7953691215344770f: No bootstrap required, opened a new log
I20260924 16:12:10.898332 31703 ts_tablet_manager.cc:1397] T 2172a7a615d441d1ae2b6dedcb08dfbf P 1b954844353e4df7953691215344770f: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.009s
I20260924 16:12:10.898332 31704 ts_tablet_manager.cc:1397] T 2172a7a615d441d1ae2b6dedcb08dfbf P 37c39763e45a47c79a38a7be18974bc9: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260924 16:12:10.902612 31702 raft_consensus.cc:348] T 2172a7a615d441d1ae2b6dedcb08dfbf P a4eeb08a503b4ce4a1bd4d3696a15668 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b954844353e4df7953691215344770f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 38325 } } peers { permanent_uuid: "a4eeb08a503b4ce4a1bd4d3696a15668" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 34251 } } peers { permanent_uuid: "37c39763e45a47c79a38a7be18974bc9" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 45941 } }
I20260924 16:12:10.903015 31702 raft_consensus.cc:374] T 2172a7a615d441d1ae2b6dedcb08dfbf P a4eeb08a503b4ce4a1bd4d3696a15668 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:10.903110 31702 raft_consensus.cc:729] T 2172a7a615d441d1ae2b6dedcb08dfbf P a4eeb08a503b4ce4a1bd4d3696a15668 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4eeb08a503b4ce4a1bd4d3696a15668, State: Initialized, Role: FOLLOWER
I20260924 16:12:10.903775 31702 consensus_queue.cc:260] T 2172a7a615d441d1ae2b6dedcb08dfbf P a4eeb08a503b4ce4a1bd4d3696a15668 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b954844353e4df7953691215344770f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 38325 } } peers { permanent_uuid: "a4eeb08a503b4ce4a1bd4d3696a15668" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 34251 } } peers { permanent_uuid: "37c39763e45a47c79a38a7be18974bc9" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 45941 } }
I20260924 16:12:10.904959 31684 heartbeater.cc:503] Master 127.29.234.62:44465 was elected leader, sending a full tablet report...
I20260924 16:12:10.905791 31702 ts_tablet_manager.cc:1428] T 2172a7a615d441d1ae2b6dedcb08dfbf P a4eeb08a503b4ce4a1bd4d3696a15668: Time spent starting tablet: real 0.012s user 0.012s sys 0.001s
I20260924 16:12:10.909590 31704 raft_consensus.cc:348] T 2172a7a615d441d1ae2b6dedcb08dfbf P 37c39763e45a47c79a38a7be18974bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b954844353e4df7953691215344770f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 38325 } } peers { permanent_uuid: "a4eeb08a503b4ce4a1bd4d3696a15668" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 34251 } } peers { permanent_uuid: "37c39763e45a47c79a38a7be18974bc9" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 45941 } }
I20260924 16:12:10.909590 31703 raft_consensus.cc:348] T 2172a7a615d441d1ae2b6dedcb08dfbf P 1b954844353e4df7953691215344770f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b954844353e4df7953691215344770f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 38325 } } peers { permanent_uuid: "a4eeb08a503b4ce4a1bd4d3696a15668" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 34251 } } peers { permanent_uuid: "37c39763e45a47c79a38a7be18974bc9" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 45941 } }
I20260924 16:12:10.910054 31704 raft_consensus.cc:374] T 2172a7a615d441d1ae2b6dedcb08dfbf P 37c39763e45a47c79a38a7be18974bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:10.910054 31703 raft_consensus.cc:374] T 2172a7a615d441d1ae2b6dedcb08dfbf P 1b954844353e4df7953691215344770f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:10.910162 31704 raft_consensus.cc:729] T 2172a7a615d441d1ae2b6dedcb08dfbf P 37c39763e45a47c79a38a7be18974bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37c39763e45a47c79a38a7be18974bc9, State: Initialized, Role: FOLLOWER
I20260924 16:12:10.910162 31703 raft_consensus.cc:729] T 2172a7a615d441d1ae2b6dedcb08dfbf P 1b954844353e4df7953691215344770f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b954844353e4df7953691215344770f, State: Initialized, Role: FOLLOWER
I20260924 16:12:10.910810 31704 consensus_queue.cc:260] T 2172a7a615d441d1ae2b6dedcb08dfbf P 37c39763e45a47c79a38a7be18974bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b954844353e4df7953691215344770f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 38325 } } peers { permanent_uuid: "a4eeb08a503b4ce4a1bd4d3696a15668" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 34251 } } peers { permanent_uuid: "37c39763e45a47c79a38a7be18974bc9" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 45941 } }
I20260924 16:12:10.910809 31703 consensus_queue.cc:260] T 2172a7a615d441d1ae2b6dedcb08dfbf P 1b954844353e4df7953691215344770f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b954844353e4df7953691215344770f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 38325 } } peers { permanent_uuid: "a4eeb08a503b4ce4a1bd4d3696a15668" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 34251 } } peers { permanent_uuid: "37c39763e45a47c79a38a7be18974bc9" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 45941 } }
I20260924 16:12:10.911913 31548 heartbeater.cc:503] Master 127.29.234.62:44465 was elected leader, sending a full tablet report...
I20260924 16:12:10.912807 31703 ts_tablet_manager.cc:1428] T 2172a7a615d441d1ae2b6dedcb08dfbf P 1b954844353e4df7953691215344770f: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260924 16:12:10.912981 31704 ts_tablet_manager.cc:1428] T 2172a7a615d441d1ae2b6dedcb08dfbf P 37c39763e45a47c79a38a7be18974bc9: Time spent starting tablet: real 0.015s user 0.014s sys 0.001s
W20260924 16:12:10.983937 31412 tablet.cc:2369] T 2172a7a615d441d1ae2b6dedcb08dfbf P 37c39763e45a47c79a38a7be18974bc9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 16:12:11.007603 31549 tablet.cc:2369] T 2172a7a615d441d1ae2b6dedcb08dfbf P 1b954844353e4df7953691215344770f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 16:12:11.016161 31685 tablet.cc:2369] T 2172a7a615d441d1ae2b6dedcb08dfbf P a4eeb08a503b4ce4a1bd4d3696a15668: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:12:11.133561 31709 raft_consensus.cc:482] T 2172a7a615d441d1ae2b6dedcb08dfbf P 1b954844353e4df7953691215344770f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:12:11.133843 31709 raft_consensus.cc:504] T 2172a7a615d441d1ae2b6dedcb08dfbf P 1b954844353e4df7953691215344770f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b954844353e4df7953691215344770f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 38325 } } peers { permanent_uuid: "a4eeb08a503b4ce4a1bd4d3696a15668" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 34251 } } peers { permanent_uuid: "37c39763e45a47c79a38a7be18974bc9" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 45941 } }
I20260924 16:12:11.135555 31709 leader_election.cc:288] T 2172a7a615d441d1ae2b6dedcb08dfbf P 1b954844353e4df7953691215344770f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4eeb08a503b4ce4a1bd4d3696a15668 (127.29.234.3:34251), 37c39763e45a47c79a38a7be18974bc9 (127.29.234.1:45941)
I20260924 16:12:11.143157 31701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:11.136156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.2:58069 (local address 127.29.234.1:45941)
0924 16:12:11.136409 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:11.136419 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:11.137633 (+ 1214us) server_negotiation.cc:407] Connection header received
0924 16:12:11.137837 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:11.137843 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:11.137915 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:11.138012 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:12:11.138666 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.139397 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.141287 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.142167 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.142371 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:11.142454 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:11.142758 (+ 304us) server_negotiation.cc:299] Negotiation successful
0924 16:12:11.142897 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 16:12:11.143309 31711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:11.135994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:45941 (local address 127.29.234.2:58069)
0924 16:12:11.137504 (+ 1510us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:11.137519 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:11.137677 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:11.138029 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:11.138034 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:11.138058 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:12:11.138495 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:11.138504 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.139525 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.139533 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:11.141113 (+ 1580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:11.141129 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.142395 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.142402 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:11.142567 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:11.142592 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 16:12:11.142709 (+ 117us) client_negotiation.cc:240] Negotiation successful
0924 16:12:11.142862 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1407,"mutex_wait_us":1,"thread_start_us":767,"threads_started":1}
I20260924 16:12:11.143420 31699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:11.136134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.2:41397 (local address 127.29.234.3:34251)
0924 16:12:11.136411 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:11.136420 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:11.137523 (+ 1103us) server_negotiation.cc:407] Connection header received
0924 16:12:11.137750 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:11.137756 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:11.137826 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:11.138116 (+ 290us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:12:11.138811 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.139628 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.141626 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.142434 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.142633 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:11.142726 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:11.143030 (+ 304us) server_negotiation.cc:299] Negotiation successful
0924 16:12:11.143167 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 16:12:11.143839 31712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:11.135883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.3:34251 (local address 127.29.234.2:41397)
0924 16:12:11.137391 (+ 1508us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:11.137418 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:11.137603 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:11.138121 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:11.138126 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:11.138230 (+ 104us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:12:11.138654 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:11.138665 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.139790 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.139796 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:11.141466 (+ 1670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:11.141479 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.142690 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.142697 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:11.142868 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:11.142891 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 16:12:11.143004 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 16:12:11.143139 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1332,"thread_start_us":1123,"threads_started":1}
I20260924 16:12:11.144500 31365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2172a7a615d441d1ae2b6dedcb08dfbf" candidate_uuid: "1b954844353e4df7953691215344770f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37c39763e45a47c79a38a7be18974bc9" is_pre_election: true
I20260924 16:12:11.144973 31638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2172a7a615d441d1ae2b6dedcb08dfbf" candidate_uuid: "1b954844353e4df7953691215344770f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4eeb08a503b4ce4a1bd4d3696a15668" is_pre_election: true
I20260924 16:12:11.145066 31365 raft_consensus.cc:2454] T 2172a7a615d441d1ae2b6dedcb08dfbf P 37c39763e45a47c79a38a7be18974bc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b954844353e4df7953691215344770f in term 0.
I20260924 16:12:11.145515 31638 raft_consensus.cc:2454] T 2172a7a615d441d1ae2b6dedcb08dfbf P a4eeb08a503b4ce4a1bd4d3696a15668 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b954844353e4df7953691215344770f in term 0.
I20260924 16:12:11.145872 31436 leader_election.cc:302] T 2172a7a615d441d1ae2b6dedcb08dfbf P 1b954844353e4df7953691215344770f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b954844353e4df7953691215344770f, 37c39763e45a47c79a38a7be18974bc9; no voters:
I20260924 16:12:11.146390 31709 raft_consensus.cc:2781] T 2172a7a615d441d1ae2b6dedcb08dfbf P 1b954844353e4df7953691215344770f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:12:11.146569 31709 raft_consensus.cc:482] T 2172a7a615d441d1ae2b6dedcb08dfbf P 1b954844353e4df7953691215344770f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:12:11.146675 31709 raft_consensus.cc:3037] T 2172a7a615d441d1ae2b6dedcb08dfbf P 1b954844353e4df7953691215344770f [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:11.148842 31709 raft_consensus.cc:504] T 2172a7a615d441d1ae2b6dedcb08dfbf P 1b954844353e4df7953691215344770f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b954844353e4df7953691215344770f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 38325 } } peers { permanent_uuid: "a4eeb08a503b4ce4a1bd4d3696a15668" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 34251 } } peers { permanent_uuid: "37c39763e45a47c79a38a7be18974bc9" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 45941 } }
I20260924 16:12:11.149511 31709 leader_election.cc:288] T 2172a7a615d441d1ae2b6dedcb08dfbf P 1b954844353e4df7953691215344770f [CANDIDATE]: Term 1 election: Requested vote from peers a4eeb08a503b4ce4a1bd4d3696a15668 (127.29.234.3:34251), 37c39763e45a47c79a38a7be18974bc9 (127.29.234.1:45941)
I20260924 16:12:11.150031 31638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2172a7a615d441d1ae2b6dedcb08dfbf" candidate_uuid: "1b954844353e4df7953691215344770f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4eeb08a503b4ce4a1bd4d3696a15668"
I20260924 16:12:11.150031 31365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2172a7a615d441d1ae2b6dedcb08dfbf" candidate_uuid: "1b954844353e4df7953691215344770f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37c39763e45a47c79a38a7be18974bc9"
I20260924 16:12:11.150264 31638 raft_consensus.cc:3037] T 2172a7a615d441d1ae2b6dedcb08dfbf P a4eeb08a503b4ce4a1bd4d3696a15668 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:11.150250 31365 raft_consensus.cc:3037] T 2172a7a615d441d1ae2b6dedcb08dfbf P 37c39763e45a47c79a38a7be18974bc9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:11.153710 31638 raft_consensus.cc:2454] T 2172a7a615d441d1ae2b6dedcb08dfbf P a4eeb08a503b4ce4a1bd4d3696a15668 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b954844353e4df7953691215344770f in term 1.
I20260924 16:12:11.153710 31365 raft_consensus.cc:2454] T 2172a7a615d441d1ae2b6dedcb08dfbf P 37c39763e45a47c79a38a7be18974bc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b954844353e4df7953691215344770f in term 1.
I20260924 16:12:11.154121 31436 leader_election.cc:302] T 2172a7a615d441d1ae2b6dedcb08dfbf P 1b954844353e4df7953691215344770f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b954844353e4df7953691215344770f, 37c39763e45a47c79a38a7be18974bc9; no voters:
I20260924 16:12:11.154439 31709 raft_consensus.cc:2781] T 2172a7a615d441d1ae2b6dedcb08dfbf P 1b954844353e4df7953691215344770f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:12:11.154865 31709 raft_consensus.cc:686] T 2172a7a615d441d1ae2b6dedcb08dfbf P 1b954844353e4df7953691215344770f [term 1 LEADER]: Becoming Leader. State: Replica: 1b954844353e4df7953691215344770f, State: Running, Role: LEADER
I20260924 16:12:11.155414 31709 consensus_queue.cc:237] T 2172a7a615d441d1ae2b6dedcb08dfbf P 1b954844353e4df7953691215344770f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b954844353e4df7953691215344770f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 38325 } } peers { permanent_uuid: "a4eeb08a503b4ce4a1bd4d3696a15668" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 34251 } } peers { permanent_uuid: "37c39763e45a47c79a38a7be18974bc9" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 45941 } }
I20260924 16:12:11.160262 31218 catalog_manager.cc:5495] T 2172a7a615d441d1ae2b6dedcb08dfbf P 1b954844353e4df7953691215344770f reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b954844353e4df7953691215344770f (127.29.234.2). New cstate: current_term: 1 leader_uuid: "1b954844353e4df7953691215344770f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b954844353e4df7953691215344770f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 38325 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4eeb08a503b4ce4a1bd4d3696a15668" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 34251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37c39763e45a47c79a38a7be18974bc9" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 45941 } health_report { overall_health: UNKNOWN } } }
Sep 24 16:12:11 dist-test-slave-v5j9 krb5kdc[31174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.234.1@KRBTEST.COM
I20260924 16:12:11.219441 31701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:11.204860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43718 (local address 127.29.234.1:45941)
0924 16:12:11.205101 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:11.205110 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:11.205750 (+ 640us) server_negotiation.cc:407] Connection header received
0924 16:12:11.205908 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:11.205912 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:11.205992 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:11.206079 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:11.207794 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.208548 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.209853 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.210284 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.210397 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:11.214382 (+ 3985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:11.214398 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:11.214415 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:11.214451 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:11.217113 (+ 2662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:11.217754 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:11.217761 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:11.217763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:11.217894 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:11.218195 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:11.218198 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:11.218200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:11.218571 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:11.218789 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:11.218991 (+ 202us) server_negotiation.cc:299] Negotiation successful
0924 16:12:11.219162 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
Sep 24 16:12:11 dist-test-slave-v5j9 krb5kdc[31174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266331, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 16:12:11 dist-test-slave-v5j9 krb5kdc[31174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266331, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.29.234.1@KRBTEST.COM
I20260924 16:12:11.268715 31701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:11.252421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43730 (local address 127.29.234.1:45941)
0924 16:12:11.252696 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:11.252710 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:11.253241 (+ 531us) server_negotiation.cc:407] Connection header received
0924 16:12:11.253467 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:11.253475 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:11.253563 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:11.253670 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:11.255040 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.256161 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.257459 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.258111 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.258230 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:11.262648 (+ 4418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:11.262654 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:11.262658 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:11.262695 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:11.266018 (+ 3323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:11.266725 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:11.266730 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:11.266735 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:11.266879 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:11.267305 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:11.267308 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:11.267310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:11.267576 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:11.267857 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:11.268125 (+ 268us) server_negotiation.cc:299] Negotiation successful
0924 16:12:11.268351 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
Sep 24 16:12:11 dist-test-slave-v5j9 krb5kdc[31174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 16:12:11 dist-test-slave-v5j9 krb5kdc[31174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.234.1@KRBTEST.COM
I20260924 16:12:11.315413 31701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:11.301078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43738 (local address 127.29.234.1:45941)
0924 16:12:11.301337 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:11.301368 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:11.301684 (+ 316us) server_negotiation.cc:407] Connection header received
0924 16:12:11.301904 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:11.301915 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:11.301983 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:11.302092 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:11.303477 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.304184 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.305495 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.305932 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.306082 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:11.310044 (+ 3962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:11.310049 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:11.310052 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:11.310092 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:11.312870 (+ 2778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:11.313551 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:11.313555 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:11.313558 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:11.313670 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:11.313984 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:11.313987 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:11.313989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:11.314267 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:11.314456 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:11.314972 (+ 516us) server_negotiation.cc:299] Negotiation successful
0924 16:12:11.315149 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
W20260924 16:12:11.317469 31325 tablet_service.cc:2908] Rejecting scan request for tablet 2172a7a615d441d1ae2b6dedcb08dfbf: Uninitialized: clean time has not yet been initialized
I20260924 16:12:11.320298 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 31282
I20260924 16:12:11.333030 31701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:11.321316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43748 (local address 127.29.234.1:45941)
0924 16:12:11.321500 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:11.321504 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:11.321880 (+ 376us) server_negotiation.cc:407] Connection header received
0924 16:12:11.322000 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:11.322004 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:11.322078 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:11.322156 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:11.323318 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.324007 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.325146 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.325632 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.325744 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:11.327946 (+ 2202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:11.327951 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:11.327954 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:11.327985 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:11.330500 (+ 2515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:11.331132 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:11.331136 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:11.331139 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:11.331282 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:11.331597 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:11.331600 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:11.331602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:11.331890 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:11.332103 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:11.332573 (+ 470us) server_negotiation.cc:299] Negotiation successful
0924 16:12:11.332771 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 16:12:11.333873 31406 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:12:11.423799 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 31282
I20260924 16:12:11.441170 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 31417
Sep 24 16:12:11 dist-test-slave-v5j9 krb5kdc[31174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.234.2@KRBTEST.COM
I20260924 16:12:11.456207 31728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:11.442011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58634 (local address 127.29.234.2:38325)
0924 16:12:11.442529 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:11.442533 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:11.442550 (+ 17us) server_negotiation.cc:407] Connection header received
0924 16:12:11.442605 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:11.442609 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:11.442674 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:11.442763 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:11.443939 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.444643 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.446038 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.446431 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.446522 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:11.450589 (+ 4067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:11.450607 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:11.450626 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:11.450666 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:11.453698 (+ 3032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:11.454367 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:11.454375 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:11.454377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:11.454511 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:11.454820 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:11.454824 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:11.454826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:11.455149 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:11.455374 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:11.455632 (+ 258us) server_negotiation.cc:299] Negotiation successful
0924 16:12:11.455801 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":287,"threads_started":1}
I20260924 16:12:11.456887 31543 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:12:11.546416 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 31417
I20260924 16:12:11.564154 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 31554
Sep 24 16:12:11 dist-test-slave-v5j9 krb5kdc[31174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.234.3@KRBTEST.COM
I20260924 16:12:11.578655 31699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:11.564929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53250 (local address 127.29.234.3:34251)
0924 16:12:11.565135 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:11.565140 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:11.565157 (+ 17us) server_negotiation.cc:407] Connection header received
0924 16:12:11.565259 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:11.565263 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:11.565334 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:11.565466 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:11.566734 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.567506 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.568685 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:11.569049 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:11.569135 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:11.573222 (+ 4087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:11.573237 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:11.573263 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:11.573298 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:11.576049 (+ 2751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:11.576782 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:11.576790 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:11.576792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:11.576977 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:11.577302 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:11.577305 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:11.577307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:11.577722 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:11.577938 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:11.578195 (+ 257us) server_negotiation.cc:299] Negotiation successful
0924 16:12:11.578391 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 16:12:11.579285 31679 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:12:11.666167 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 31554
I20260924 16:12:11.683239 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 31187
I20260924 16:12:11.684450 31250 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:12:11.792622 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 31187
2026-09-24T16:12:11Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4386 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:12:11 dist-test-slave-v5j9 krb5kdc[31734](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:12:11 dist-test-slave-v5j9 krb5kdc[31734](info): set up 2 sockets
Sep 24 16:12:11 dist-test-slave-v5j9 krb5kdc[31734](info): commencing operation
krb5kdc: starting...
W20260924 16:12:12.876932 30632 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Sep 24 16:12:12 dist-test-slave-v5j9 krb5kdc[31734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:12:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:12:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.234.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab.
Entry for principal kudu/127.29.234.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab.
I20260924 16:12:13.065655 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:39697
--webserver_interface=127.29.234.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab
--principal=kudu/127.29.234.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:37949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.234.62:39697
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:12:13.394026 31747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:13.394371 31747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:13.394529 31747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:13.401924 31747 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:12:13.402040 31747 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 16:12:13.402112 31747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:13.402226 31747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:13.402307 31747 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:12:13.402417 31747 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:12:13.412319 31747 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:37949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.234.62:39697
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab
--principal=kudu/127.29.234.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:39697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.234.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:12:13.413709 31747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:13.415062 31747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:13.427433 31755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:13.427477 31752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:13.427860 31753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:13.428277 31747 server_base.cc:1034] running on GCE node
I20260924 16:12:13.429028 31747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:13.430599 31747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:13.431840 31747 hybrid_clock.cc:648] HybridClock initialized: now 1790266333431777 us; error 41 us; skew 500 ppm
I20260924 16:12:13.432204 31747 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:12:13 dist-test-slave-v5j9 krb5kdc[31734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266333, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:13.435635 31747 init.cc:377] Logged in from keytab as kudu/127.29.234.62@KRBTEST.COM (short username kudu)
I20260924 16:12:13.437291 31747 webserver.cc:460] Webserver started at http://127.29.234.62:33351/ using document root <none> and password file <none>
I20260924 16:12:13.438148 31747 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:13.438289 31747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:13.438627 31747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:13.441740 31747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/minicluster-data/master-0/data/instance:
uuid: "dde4f26be06a4b1799e168313c156528"
format_stamp: "Formatted at 2026-09-24 16:12:13 on dist-test-slave-v5j9"
server_key: "d798f2b5320332b7e58f92fbd0285630"
server_key_iv: "98b717240fabb9d9303226c26f87c7ec"
server_key_version: "encryptionkey@0"
I20260924 16:12:13.442523 31747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/minicluster-data/master-0/wal/instance:
uuid: "dde4f26be06a4b1799e168313c156528"
format_stamp: "Formatted at 2026-09-24 16:12:13 on dist-test-slave-v5j9"
server_key: "d798f2b5320332b7e58f92fbd0285630"
server_key_iv: "98b717240fabb9d9303226c26f87c7ec"
server_key_version: "encryptionkey@0"
I20260924 16:12:13.447808 31747 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260924 16:12:13.451285 31762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:13.452616 31747 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 16:12:13.452805 31747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/minicluster-data/master-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/minicluster-data/master-0/wal
uuid: "dde4f26be06a4b1799e168313c156528"
format_stamp: "Formatted at 2026-09-24 16:12:13 on dist-test-slave-v5j9"
server_key: "d798f2b5320332b7e58f92fbd0285630"
server_key_iv: "98b717240fabb9d9303226c26f87c7ec"
server_key_version: "encryptionkey@0"
I20260924 16:12:13.453032 31747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:13.487732 31747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:13.491828 31747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:13.492117 31747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:13.513926 31747 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.62:39697
I20260924 16:12:13.513947 31824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.62:39697 every 8 connection(s)
I20260924 16:12:13.515372 31747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
I20260924 16:12:13.519289 31825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:13.524698 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 31747
I20260924 16:12:13.524922 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/minicluster-data/master-0/wal/instance
I20260924 16:12:13.525774 30632 external_mini_cluster.cc:1428] Setting key fdb2d89f1829189dcfa5b8d1fa027c1a
I20260924 16:12:13.531193 31825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dde4f26be06a4b1799e168313c156528: Bootstrap starting.
I20260924 16:12:13.535092 31825 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dde4f26be06a4b1799e168313c156528: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:13.536357 31825 log.cc:824] T 00000000000000000000000000000000 P dde4f26be06a4b1799e168313c156528: Log is configured to *not* fsync() on all Append() calls
I20260924 16:12:13.539762 31825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dde4f26be06a4b1799e168313c156528: No bootstrap required, opened a new log
Sep 24 16:12:13 dist-test-slave-v5j9 krb5kdc[31734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:12:13.549613 31825 raft_consensus.cc:348] T 00000000000000000000000000000000 P dde4f26be06a4b1799e168313c156528 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dde4f26be06a4b1799e168313c156528" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 39697 } }
I20260924 16:12:13.550109 31825 raft_consensus.cc:374] T 00000000000000000000000000000000 P dde4f26be06a4b1799e168313c156528 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:13.550227 31825 raft_consensus.cc:729] T 00000000000000000000000000000000 P dde4f26be06a4b1799e168313c156528 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dde4f26be06a4b1799e168313c156528, State: Initialized, Role: FOLLOWER
I20260924 16:12:13.551023 31825 consensus_queue.cc:260] T 00000000000000000000000000000000 P dde4f26be06a4b1799e168313c156528 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dde4f26be06a4b1799e168313c156528" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 39697 } }
I20260924 16:12:13.551342 31825 raft_consensus.cc:388] T 00000000000000000000000000000000 P dde4f26be06a4b1799e168313c156528 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:12:13.551484 31825 raft_consensus.cc:482] T 00000000000000000000000000000000 P dde4f26be06a4b1799e168313c156528 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:12:13.551671 31825 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dde4f26be06a4b1799e168313c156528 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:13.552215 31828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:13.528332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50724 (local address 127.29.234.62:39697)
0924 16:12:13.529405 (+ 1073us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:13.529421 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:13.529796 (+ 375us) server_negotiation.cc:407] Connection header received
0924 16:12:13.532236 (+ 2440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:13.532264 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:13.532745 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:13.533248 (+ 503us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:13.534531 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:13.536584 (+ 2053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:13.538284 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:13.539326 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:13.539518 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:13.543501 (+ 3983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:13.543512 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:13.543535 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:13.543604 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:13.547637 (+ 4033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:13.548303 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:13.548313 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:13.548322 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:13.548555 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:13.548884 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:13.548889 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:13.548892 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:13.549640 (+ 748us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:13.549926 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:13.550285 (+ 359us) server_negotiation.cc:299] Negotiation successful
0924 16:12:13.550750 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":761,"thread_start_us":467,"threads_started":1}
I20260924 16:12:13.554297 31825 raft_consensus.cc:504] T 00000000000000000000000000000000 P dde4f26be06a4b1799e168313c156528 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dde4f26be06a4b1799e168313c156528" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 39697 } }
I20260924 16:12:13.554831 31825 leader_election.cc:302] T 00000000000000000000000000000000 P dde4f26be06a4b1799e168313c156528 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dde4f26be06a4b1799e168313c156528; no voters:
I20260924 16:12:13.555518 31825 leader_election.cc:288] T 00000000000000000000000000000000 P dde4f26be06a4b1799e168313c156528 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:12:13.555612 31830 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dde4f26be06a4b1799e168313c156528 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:12:13.556445 31830 raft_consensus.cc:686] T 00000000000000000000000000000000 P dde4f26be06a4b1799e168313c156528 [term 1 LEADER]: Becoming Leader. State: Replica: dde4f26be06a4b1799e168313c156528, State: Running, Role: LEADER
I20260924 16:12:13.557014 31830 consensus_queue.cc:237] T 00000000000000000000000000000000 P dde4f26be06a4b1799e168313c156528 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dde4f26be06a4b1799e168313c156528" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 39697 } }
I20260924 16:12:13.557941 31825 sys_catalog.cc:564] T 00000000000000000000000000000000 P dde4f26be06a4b1799e168313c156528 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:12:13.560988 31832 sys_catalog.cc:455] T 00000000000000000000000000000000 P dde4f26be06a4b1799e168313c156528 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dde4f26be06a4b1799e168313c156528. Latest consensus state: current_term: 1 leader_uuid: "dde4f26be06a4b1799e168313c156528" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dde4f26be06a4b1799e168313c156528" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 39697 } } }
I20260924 16:12:13.561533 31832 sys_catalog.cc:458] T 00000000000000000000000000000000 P dde4f26be06a4b1799e168313c156528 [sys.catalog]: This master's current role is: LEADER
I20260924 16:12:13.565490 31831 sys_catalog.cc:455] T 00000000000000000000000000000000 P dde4f26be06a4b1799e168313c156528 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dde4f26be06a4b1799e168313c156528" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dde4f26be06a4b1799e168313c156528" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 39697 } } }
I20260924 16:12:13.565745 31831 sys_catalog.cc:458] T 00000000000000000000000000000000 P dde4f26be06a4b1799e168313c156528 [sys.catalog]: This master's current role is: LEADER
I20260924 16:12:13.566309 31839 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:12:13.571822 31839 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:12:13.580410 31839 catalog_manager.cc:1347] Generated new cluster ID: 19a8abb1af64422a83ac13d008644b1a
I20260924 16:12:13.580538 31839 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:12:13.601414 31839 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:12:13.603097 31839 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:12:13.609953 31839 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dde4f26be06a4b1799e168313c156528: Generated new TSK 0
I20260924 16:12:13.611094 31839 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.234.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab.
Entry for principal kudu/127.29.234.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab.
I20260924 16:12:13.658038 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.1:0
--local_ip_for_outbound_sockets=127.29.234.1
--webserver_interface=127.29.234.1
--webserver_port=0
--tserver_master_addrs=127.29.234.62:39697
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab
--principal=kudu/127.29.234.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:37949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:12:13.992969 31851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:13.993325 31851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:13.993543 31851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:14.001588 31851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:14.001724 31851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:14.001874 31851 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 16:12:14.002017 31851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.1
I20260924 16:12:14.012746 31851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:37949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab
--principal=kudu/127.29.234.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.234.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.234.62:39697
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.1
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:12:14.014257 31851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:14.015648 31851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:14.028682 31856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:14.028656 31857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:14.028795 31859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:14.029449 31851 server_base.cc:1034] running on GCE node
I20260924 16:12:14.030048 31851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:14.030944 31851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:14.032248 31851 hybrid_clock.cc:648] HybridClock initialized: now 1790266334032165 us; error 48 us; skew 500 ppm
I20260924 16:12:14.032618 31851 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:12:14 dist-test-slave-v5j9 krb5kdc[31734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266334, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:14.036020 31851 init.cc:377] Logged in from keytab as kudu/127.29.234.1@KRBTEST.COM (short username kudu)
I20260924 16:12:14.037716 31851 webserver.cc:460] Webserver started at http://127.29.234.1:40533/ using document root <none> and password file <none>
I20260924 16:12:14.038462 31851 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:14.038614 31851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:14.038955 31851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:14.042043 31851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/minicluster-data/ts-0/data/instance:
uuid: "f81b244edb0c46c38601587481b42ef0"
format_stamp: "Formatted at 2026-09-24 16:12:14 on dist-test-slave-v5j9"
server_key: "1e734158607ae99e3a0a63d00bb2922d"
server_key_iv: "5781361304a40eeaa43cd9de69aaf3c9"
server_key_version: "encryptionkey@0"
I20260924 16:12:14.042860 31851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/minicluster-data/ts-0/wal/instance:
uuid: "f81b244edb0c46c38601587481b42ef0"
format_stamp: "Formatted at 2026-09-24 16:12:14 on dist-test-slave-v5j9"
server_key: "1e734158607ae99e3a0a63d00bb2922d"
server_key_iv: "5781361304a40eeaa43cd9de69aaf3c9"
server_key_version: "encryptionkey@0"
I20260924 16:12:14.048012 31851 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 16:12:14.051582 31867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:14.052887 31851 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 16:12:14.053071 31851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/minicluster-data/ts-0/wal
uuid: "f81b244edb0c46c38601587481b42ef0"
format_stamp: "Formatted at 2026-09-24 16:12:14 on dist-test-slave-v5j9"
server_key: "1e734158607ae99e3a0a63d00bb2922d"
server_key_iv: "5781361304a40eeaa43cd9de69aaf3c9"
server_key_version: "encryptionkey@0"
I20260924 16:12:14.053299 31851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:14.067217 31851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:14.071674 31851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:14.071966 31851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:14.075104 31851 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:12:14.075222 31851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:14.075389 31851 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:12:14.075557 31851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Sep 24 16:12:14 dist-test-slave-v5j9 krb5kdc[31734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266334, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.1@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:12:14.097062 31875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:14.075185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.1:46021 (local address 127.29.234.62:39697)
0924 16:12:14.075659 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:14.075666 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:14.076942 (+ 1276us) server_negotiation.cc:407] Connection header received
0924 16:12:14.079082 (+ 2140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:14.079087 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:14.079179 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:14.079295 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:14.081314 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:14.082406 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:14.084974 (+ 2568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:14.085592 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:14.085725 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:14.090081 (+ 4356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:14.090090 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:14.090098 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:14.090146 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:14.093266 (+ 3120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:14.094538 (+ 1272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:14.094548 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:14.094551 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:14.094674 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:14.095188 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:14.095195 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:14.095201 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:14.095554 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:14.095710 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:14.096439 (+ 729us) server_negotiation.cc:299] Negotiation successful
0924 16:12:14.096694 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":235,"threads_started":1}
I20260924 16:12:14.098654 31877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:14.075829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:39697 (local address 127.29.234.1:46021)
0924 16:12:14.076752 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:14.076802 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:14.078796 (+ 1994us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:14.079542 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:14.079549 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:14.080188 (+ 639us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:14.081131 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:14.081150 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:14.082574 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:14.082585 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:14.084817 (+ 2232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:14.084829 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:14.085821 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:14.085831 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:14.086057 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:14.086816 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:14.086836 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:14.089820 (+ 2984us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:14.093524 (+ 3704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:14.093536 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:14.093556 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:14.094360 (+ 804us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:14.094797 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:14.094801 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:14.094805 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:14.095004 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:14.095700 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:14.095708 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:14.096070 (+ 362us) client_negotiation.cc:769] Sending connection context
0924 16:12:14.096359 (+ 289us) client_negotiation.cc:240] Negotiation successful
0924 16:12:14.096897 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":416,"threads_started":1}
I20260924 16:12:14.119740 31875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:14.104902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.1:53347 (local address 127.29.234.62:39697)
0924 16:12:14.105094 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:14.105102 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:14.105968 (+ 866us) server_negotiation.cc:407] Connection header received
0924 16:12:14.106186 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:14.106194 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:14.106291 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:14.106399 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:14.107902 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:14.108647 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:14.110351 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:14.110760 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:14.110910 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:14.112814 (+ 1904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:14.112824 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:14.112833 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:14.112885 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:14.116761 (+ 3876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:14.117487 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:14.117496 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:14.117502 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:14.117666 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:14.118144 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:14.118148 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:14.118157 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:14.118463 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:14.118653 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:14.119255 (+ 602us) server_negotiation.cc:299] Negotiation successful
0924 16:12:14.119502 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 16:12:14.120229 31932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:14.105029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:39697 (local address 127.29.234.1:53347)
0924 16:12:14.105866 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:14.105882 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:14.106040 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:14.106458 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:14.106472 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:14.107212 (+ 740us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:14.107693 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:14.107710 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:14.108781 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:14.108787 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:14.110203 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:14.110216 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:14.110959 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:14.110965 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:14.111154 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:14.111829 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:14.111845 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:14.112570 (+ 725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:14.116898 (+ 4328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:14.116902 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:14.116905 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:14.117327 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:14.117798 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:14.117805 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:14.117811 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:14.117995 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:14.118589 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:14.118618 (+ 29us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:14.118906 (+ 288us) client_negotiation.cc:769] Sending connection context
0924 16:12:14.119105 (+ 199us) client_negotiation.cc:240] Negotiation successful
0924 16:12:14.119567 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"thread_start_us":577,"threads_started":1}
I20260924 16:12:14.133314 31851 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.1:35217
I20260924 16:12:14.133386 31988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.1:35217 every 8 connection(s)
I20260924 16:12:14.134622 31851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
I20260924 16:12:14.136829 31989 heartbeater.cc:347] Connected to a master server at 127.29.234.62:39697
I20260924 16:12:14.137072 31989 heartbeater.cc:464] Registering TS with master...
I20260924 16:12:14.137811 31989 heartbeater.cc:511] Master 127.29.234.62:39697 requested a full tablet report, sending...
I20260924 16:12:14.138834 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 31851
I20260924 16:12:14.139056 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/minicluster-data/ts-0/wal/instance
I20260924 16:12:14.139676 30632 external_mini_cluster.cc:1428] Setting key 34596b724a50c3b4102049fa2198b807
I20260924 16:12:14.139915 31779 ts_manager.cc:194] Registered new tserver with Master: f81b244edb0c46c38601587481b42ef0 (127.29.234.1:35217)
I20260924 16:12:14.141824 31779 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.234.1@KRBTEST.COM'} at 127.29.234.1:46021
WARNING: no policy specified for kudu/127.29.234.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab.
Entry for principal kudu/127.29.234.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab.
I20260924 16:12:14.178297 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.2:0
--local_ip_for_outbound_sockets=127.29.234.2
--webserver_interface=127.29.234.2
--webserver_port=0
--tserver_master_addrs=127.29.234.62:39697
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab
--principal=kudu/127.29.234.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:37949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:12:14.501573 31993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:14.501919 31993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:14.502095 31993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:14.509565 31993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:14.509733 31993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:14.509883 31993 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 16:12:14.510033 31993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.2
I20260924 16:12:14.520246 31993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:37949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab
--principal=kudu/127.29.234.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.234.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.234.62:39697
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.2
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:12:14.521637 31993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:14.523028 31993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:14.536531 31998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:14.536507 31999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:14.536899 32001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:14.537266 31993 server_base.cc:1034] running on GCE node
I20260924 16:12:14.537930 31993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:14.538851 31993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:14.540067 31993 hybrid_clock.cc:648] HybridClock initialized: now 1790266334539999 us; error 42 us; skew 500 ppm
I20260924 16:12:14.540378 31993 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:12:14 dist-test-slave-v5j9 krb5kdc[31734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266334, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:14.543422 31993 init.cc:377] Logged in from keytab as kudu/127.29.234.2@KRBTEST.COM (short username kudu)
I20260924 16:12:14.544997 31993 webserver.cc:460] Webserver started at http://127.29.234.2:34275/ using document root <none> and password file <none>
I20260924 16:12:14.545823 31993 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:14.545957 31993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:14.546312 31993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:14.549258 31993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/minicluster-data/ts-1/data/instance:
uuid: "fb09133e8ecc409ba9ffc6044c90180f"
format_stamp: "Formatted at 2026-09-24 16:12:14 on dist-test-slave-v5j9"
server_key: "67572a87bcb74c531615628788879ac9"
server_key_iv: "06b49a58d003e7cd632491a0a8f7c778"
server_key_version: "encryptionkey@0"
I20260924 16:12:14.550244 31993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/minicluster-data/ts-1/wal/instance:
uuid: "fb09133e8ecc409ba9ffc6044c90180f"
format_stamp: "Formatted at 2026-09-24 16:12:14 on dist-test-slave-v5j9"
server_key: "67572a87bcb74c531615628788879ac9"
server_key_iv: "06b49a58d003e7cd632491a0a8f7c778"
server_key_version: "encryptionkey@0"
I20260924 16:12:14.555605 31993 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 16:12:14.560022 32009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:14.561578 31993 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:12:14.561769 31993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/minicluster-data/ts-1/wal
uuid: "fb09133e8ecc409ba9ffc6044c90180f"
format_stamp: "Formatted at 2026-09-24 16:12:14 on dist-test-slave-v5j9"
server_key: "67572a87bcb74c531615628788879ac9"
server_key_iv: "06b49a58d003e7cd632491a0a8f7c778"
server_key_version: "encryptionkey@0"
I20260924 16:12:14.562013 31993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:14.577155 31993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:14.581231 31993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:14.581547 31993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:14.584225 31993 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:12:14.584334 31993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:14.584501 31993 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:12:14.584569 31993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 16:12:14 dist-test-slave-v5j9 krb5kdc[31734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266334, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.2@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:12:14.610060 31875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:14.584666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.2:36157 (local address 127.29.234.62:39697)
0924 16:12:14.584874 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:14.584880 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:14.586460 (+ 1580us) server_negotiation.cc:407] Connection header received
0924 16:12:14.588567 (+ 2107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:14.588577 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:14.588708 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:14.588836 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:14.591276 (+ 2440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:14.592341 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:14.595819 (+ 3478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:14.596283 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:14.596393 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:14.601583 (+ 5190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:14.601589 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:14.601592 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:14.601630 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:14.605682 (+ 4052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:14.606822 (+ 1140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:14.606828 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:14.606833 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:14.607030 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:14.607699 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:14.607705 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:14.607709 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:14.608158 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:14.608341 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:14.609167 (+ 826us) server_negotiation.cc:299] Negotiation successful
0924 16:12:14.609509 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260924 16:12:14.610908 32018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:14.585261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:39697 (local address 127.29.234.2:36157)
0924 16:12:14.586233 (+ 972us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:14.586292 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:14.588238 (+ 1946us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:14.589131 (+ 893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:14.589144 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:14.589969 (+ 825us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:14.591059 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:14.591091 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:14.592552 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:14.592562 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:14.595585 (+ 3023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:14.595611 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:14.596576 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:14.596591 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:14.597008 (+ 417us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:14.598159 (+ 1151us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:14.598186 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:14.601262 (+ 3076us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:14.605886 (+ 4624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:14.605898 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:14.605921 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:14.606674 (+ 753us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:14.607191 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:14.607201 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:14.607204 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:14.607542 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:14.608333 (+ 791us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:14.608345 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:14.608731 (+ 386us) client_negotiation.cc:769] Sending connection context
0924 16:12:14.608979 (+ 248us) client_negotiation.cc:240] Negotiation successful
0924 16:12:14.609498 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"thread_start_us":386,"threads_started":1}
I20260924 16:12:14.641002 31875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:14.624728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.2:39383 (local address 127.29.234.62:39697)
0924 16:12:14.625002 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:14.625012 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:14.625626 (+ 614us) server_negotiation.cc:407] Connection header received
0924 16:12:14.625867 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:14.625876 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:14.625968 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:14.626059 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:14.627593 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:14.628300 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:14.630851 (+ 2551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:14.631470 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:14.631625 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:14.634420 (+ 2795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:14.634428 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:14.634438 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:14.634473 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:14.637626 (+ 3153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:14.638653 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:14.638661 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:14.638666 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:14.638843 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:14.639326 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:14.639331 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:14.639334 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:14.639809 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:14.639983 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:14.640408 (+ 425us) server_negotiation.cc:299] Negotiation successful
0924 16:12:14.640638 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260924 16:12:14.641708 32096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:14.624688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:39697 (local address 127.29.234.2:39383)
0924 16:12:14.625478 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:14.625504 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:14.625711 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:14.626130 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:14.626138 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:14.626919 (+ 781us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:14.627401 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:14.627420 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:14.628459 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:14.628467 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:14.630645 (+ 2178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:14.630666 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:14.631916 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:14.631927 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:14.632219 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:14.632974 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:14.632995 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:14.634202 (+ 1207us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:14.637769 (+ 3567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:14.637774 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:14.637778 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:14.638503 (+ 725us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:14.639013 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:14.639023 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:14.639028 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:14.639194 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:14.639936 (+ 742us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:14.639941 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:14.640147 (+ 206us) client_negotiation.cc:769] Sending connection context
0924 16:12:14.640306 (+ 159us) client_negotiation.cc:240] Negotiation successful
0924 16:12:14.640696 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":382,"threads_started":1}
I20260924 16:12:14.644233 31993 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.2:44825
I20260924 16:12:14.644246 32130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.2:44825 every 8 connection(s)
I20260924 16:12:14.645871 31993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
I20260924 16:12:14.647921 32131 heartbeater.cc:347] Connected to a master server at 127.29.234.62:39697
I20260924 16:12:14.648162 32131 heartbeater.cc:464] Registering TS with master...
I20260924 16:12:14.648816 32131 heartbeater.cc:511] Master 127.29.234.62:39697 requested a full tablet report, sending...
I20260924 16:12:14.650154 31779 ts_manager.cc:194] Registered new tserver with Master: fb09133e8ecc409ba9ffc6044c90180f (127.29.234.2:44825)
I20260924 16:12:14.650903 31779 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.234.2@KRBTEST.COM'} at 127.29.234.2:36157
I20260924 16:12:14.656180 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 31993
I20260924 16:12:14.656333 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/minicluster-data/ts-1/wal/instance
I20260924 16:12:14.656831 30632 external_mini_cluster.cc:1428] Setting key 4d7d00ad969d66793c3f48ada2adb0e3
WARNING: no policy specified for kudu/127.29.234.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab.
Entry for principal kudu/127.29.234.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab.
I20260924 16:12:14.697117 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.3:0
--local_ip_for_outbound_sockets=127.29.234.3
--webserver_interface=127.29.234.3
--webserver_port=0
--tserver_master_addrs=127.29.234.62:39697
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab
--principal=kudu/127.29.234.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:37949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:12:15.023594 32135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:15.023929 32135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:15.024111 32135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:15.032071 32135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:15.032248 32135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:15.032397 32135 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 16:12:15.032544 32135 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.3
I20260924 16:12:15.042830 32135 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:37949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab
--principal=kudu/127.29.234.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.234.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.234.62:39697
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.3
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:12:15.044194 32135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:15.045586 32135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:15.059453 32142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:15.059476 32141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:15.060290 32144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:15.060441 32135 server_base.cc:1034] running on GCE node
I20260924 16:12:15.061153 32135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:15.062351 32135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:15.063611 32135 hybrid_clock.cc:648] HybridClock initialized: now 1790266335063544 us; error 34 us; skew 500 ppm
I20260924 16:12:15.063948 32135 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:12:15 dist-test-slave-v5j9 krb5kdc[31734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266335, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:15.067836 32135 init.cc:377] Logged in from keytab as kudu/127.29.234.3@KRBTEST.COM (short username kudu)
I20260924 16:12:15.069526 32135 webserver.cc:460] Webserver started at http://127.29.234.3:39323/ using document root <none> and password file <none>
I20260924 16:12:15.070284 32135 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:15.070432 32135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:15.070778 32135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:15.074002 32135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/minicluster-data/ts-2/data/instance:
uuid: "ba716e8ff56743de8c3094b708c833a7"
format_stamp: "Formatted at 2026-09-24 16:12:15 on dist-test-slave-v5j9"
server_key: "fa69cdbd4f42875ef70bb82379c62773"
server_key_iv: "34e72e331572f07753a5fdabff2ce2fd"
server_key_version: "encryptionkey@0"
I20260924 16:12:15.074836 32135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/minicluster-data/ts-2/wal/instance:
uuid: "ba716e8ff56743de8c3094b708c833a7"
format_stamp: "Formatted at 2026-09-24 16:12:15 on dist-test-slave-v5j9"
server_key: "fa69cdbd4f42875ef70bb82379c62773"
server_key_iv: "34e72e331572f07753a5fdabff2ce2fd"
server_key_version: "encryptionkey@0"
I20260924 16:12:15.080176 32135 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260924 16:12:15.083719 32151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:15.085120 32135 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 16:12:15.085309 32135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/minicluster-data/ts-2/wal
uuid: "ba716e8ff56743de8c3094b708c833a7"
format_stamp: "Formatted at 2026-09-24 16:12:15 on dist-test-slave-v5j9"
server_key: "fa69cdbd4f42875ef70bb82379c62773"
server_key_iv: "34e72e331572f07753a5fdabff2ce2fd"
server_key_version: "encryptionkey@0"
I20260924 16:12:15.085601 32135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:15.109084 32135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:15.114149 32135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:15.114455 32135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:15.117159 32135 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:12:15.117273 32135 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:15.117496 32135 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:12:15.117578 32135 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 16:12:15 dist-test-slave-v5j9 krb5kdc[31734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266335, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.3@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:12:15.142700 31875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:15.118296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.3:53495 (local address 127.29.234.62:39697)
0924 16:12:15.118470 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:15.118475 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:15.120033 (+ 1558us) server_negotiation.cc:407] Connection header received
0924 16:12:15.122198 (+ 2165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:15.122205 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:15.122315 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:15.122458 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:15.125039 (+ 2581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.126155 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.128795 (+ 2640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.129315 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.129481 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:15.134778 (+ 5297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:15.134787 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:15.134792 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:15.134837 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:15.138584 (+ 3747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:15.139735 (+ 1151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:15.139740 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:15.139742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:15.139895 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:15.140467 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:15.140470 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:15.140472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:15.140743 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:15.140866 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:15.142047 (+ 1181us) server_negotiation.cc:299] Negotiation successful
0924 16:12:15.142283 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 16:12:15.144465 32160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:15.118969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:39697 (local address 127.29.234.3:53495)
0924 16:12:15.119872 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:15.119910 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:15.121821 (+ 1911us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:15.122775 (+ 954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:15.122789 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:15.123761 (+ 972us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:15.124844 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:15.124868 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.126333 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.126341 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:15.128646 (+ 2305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:15.128658 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.129638 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.129646 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:15.130077 (+ 431us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:15.131028 (+ 951us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:15.131080 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:15.134494 (+ 3414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:15.138747 (+ 4253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:15.138761 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:15.138784 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:15.139572 (+ 788us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:15.140030 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:15.140038 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:15.140044 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:15.140340 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:15.140911 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:15.140923 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:15.141486 (+ 563us) client_negotiation.cc:769] Sending connection context
0924 16:12:15.141892 (+ 406us) client_negotiation.cc:240] Negotiation successful
0924 16:12:15.142597 (+ 705us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"thread_start_us":381,"threads_started":1}
I20260924 16:12:15.144534 31989 heartbeater.cc:503] Master 127.29.234.62:39697 was elected leader, sending a full tablet report...
I20260924 16:12:15.173609 32272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.3:39383 every 8 connection(s)
I20260924 16:12:15.173636 32135 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.3:39383
I20260924 16:12:15.175282 32135 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
I20260924 16:12:15.176676 31875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:15.161150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.3:36931 (local address 127.29.234.62:39697)
0924 16:12:15.161414 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:15.161420 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:15.161634 (+ 214us) server_negotiation.cc:407] Connection header received
0924 16:12:15.161923 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:15.161929 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:15.162024 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:15.162160 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:15.163741 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.164555 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.167121 (+ 2566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.167776 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.167910 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:15.170042 (+ 2132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:15.170051 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:15.170060 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:15.170112 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:15.173902 (+ 3790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:15.174564 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:15.174570 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:15.174575 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:15.174729 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:15.175107 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:15.175112 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:15.175116 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:15.175481 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:15.175634 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:15.176067 (+ 433us) server_negotiation.cc:299] Negotiation successful
0924 16:12:15.176306 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 16:12:15.176815 32243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:15.160651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:39697 (local address 127.29.234.3:36931)
0924 16:12:15.161530 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:15.161545 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:15.161748 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:15.162354 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:15.162359 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:15.163135 (+ 776us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:15.163559 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:15.163581 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.164759 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.164763 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:15.166834 (+ 2071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:15.166850 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.168000 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.168007 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:15.168156 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:15.168892 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:15.168910 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:15.169800 (+ 890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:15.174030 (+ 4230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:15.174033 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:15.174036 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:15.174424 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:15.174838 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:15.174841 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:15.174843 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:15.174976 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:15.175599 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:15.175604 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:15.175797 (+ 193us) client_negotiation.cc:769] Sending connection context
0924 16:12:15.175948 (+ 151us) client_negotiation.cc:240] Negotiation successful
0924 16:12:15.176339 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":339,"threads_started":1}
I20260924 16:12:15.178064 32273 heartbeater.cc:347] Connected to a master server at 127.29.234.62:39697
I20260924 16:12:15.178318 32273 heartbeater.cc:464] Registering TS with master...
I20260924 16:12:15.179070 32273 heartbeater.cc:511] Master 127.29.234.62:39697 requested a full tablet report, sending...
I20260924 16:12:15.180475 31779 ts_manager.cc:194] Registered new tserver with Master: ba716e8ff56743de8c3094b708c833a7 (127.29.234.3:39383)
I20260924 16:12:15.181565 31779 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.234.3@KRBTEST.COM'} at 127.29.234.3:53495
I20260924 16:12:15.185556 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 32135
I20260924 16:12:15.185710 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/minicluster-data/ts-2/wal/instance
I20260924 16:12:15.186120 30632 external_mini_cluster.cc:1428] Setting key d043e7976568ad74dd21920953ec0d59
I20260924 16:12:15.188822 30632 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 16:12:15 dist-test-slave-v5j9 krb5kdc[31734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266335, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 16:12:15 dist-test-slave-v5j9 krb5kdc[31734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266335, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:12:15.238546 31875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:15.221032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50730 (local address 127.29.234.62:39697)
0924 16:12:15.221297 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:15.221305 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:15.221544 (+ 239us) server_negotiation.cc:407] Connection header received
0924 16:12:15.221738 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:15.221742 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:15.221825 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:15.221915 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:15.223310 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.223988 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.225404 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.225817 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.225907 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:15.233518 (+ 7611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:15.233524 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:15.233527 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:15.233563 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:15.236019 (+ 2456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:15.236650 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:15.236654 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:15.236657 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:15.236809 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:15.237114 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:15.237118 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:15.237120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:15.237507 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:15.237696 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:15.237942 (+ 246us) server_negotiation.cc:299] Negotiation successful
0924 16:12:15.238112 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 16:12:15.242800 31779 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50730:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:12:15.249956 31779 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:12:15.282895 32288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:15.262603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58496 (local address 127.29.234.1:35217)
0924 16:12:15.273946 (+ 11343us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:15.273956 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:15.273990 (+ 34us) server_negotiation.cc:407] Connection header received
0924 16:12:15.274102 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:15.274111 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:15.274402 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:15.274623 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:12:15.275699 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.277152 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.279897 (+ 2745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.281403 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.281542 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:15.282302 (+ 760us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:15.282398 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 16:12:15.282568 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11188,"thread_start_us":11068,"threads_started":1}
I20260924 16:12:15.283210 32290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:15.262439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:35217 (local address 127.0.0.1:58496)
0924 16:12:15.272790 (+ 10351us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:15.272827 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:15.273024 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:15.274911 (+ 1887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:15.274921 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:15.274956 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:12:15.275460 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:15.275474 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.277296 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.277301 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:15.279722 (+ 2421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:15.279738 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.281609 (+ 1871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.281616 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:15.281811 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:15.281877 (+ 66us) client_negotiation.cc:769] Sending connection context
0924 16:12:15.281995 (+ 118us) client_negotiation.cc:240] Negotiation successful
0924 16:12:15.282201 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10064,"mutex_wait_us":24,"thread_start_us":12512,"threads_started":1}
I20260924 16:12:15.283545 32294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:15.263851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50374 (local address 127.29.234.2:44825)
0924 16:12:15.273508 (+ 9657us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:15.273522 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:15.275398 (+ 1876us) server_negotiation.cc:407] Connection header received
0924 16:12:15.275690 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:15.275698 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:15.275939 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:15.276107 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:12:15.276978 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.278107 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.280620 (+ 2513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.281802 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.281933 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:15.282129 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:15.282822 (+ 693us) server_negotiation.cc:299] Negotiation successful
0924 16:12:15.282995 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9409,"thread_start_us":9271,"threads_started":1}
I20260924 16:12:15.283818 32289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:15.262795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.3:39383 (local address 127.0.0.1:42636)
0924 16:12:15.274841 (+ 12046us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:15.274868 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:15.274986 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:15.275980 (+ 994us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:15.275984 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:15.276008 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:12:15.276407 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:15.276423 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.278462 (+ 2039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.278468 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:15.280716 (+ 2248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:15.280730 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.282802 (+ 2072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.282808 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:15.283036 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:15.283075 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 16:12:15.283281 (+ 206us) client_negotiation.cc:240] Negotiation successful
0924 16:12:15.283437 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11916,"mutex_wait_us":1,"spinlock_wait_cycles":1280,"thread_start_us":11754,"threads_started":1}
I20260924 16:12:15.284192 32287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:15.263053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.2:44825 (local address 127.0.0.1:50374)
0924 16:12:15.275282 (+ 12229us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:15.275299 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:15.275501 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:15.276345 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:15.276348 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:15.276372 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:12:15.276780 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:15.276795 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.278282 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.278287 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:15.280455 (+ 2168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:15.280470 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.282286 (+ 1816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.282292 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:15.282443 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:15.282473 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 16:12:15.282560 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 16:12:15.282691 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12097,"mutex_wait_us":84,"thread_start_us":9298,"threads_started":1}
I20260924 16:12:15.284474 32291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:15.263413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42636 (local address 127.29.234.3:39383)
0924 16:12:15.274376 (+ 10963us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:15.274384 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:15.275132 (+ 748us) server_negotiation.cc:407] Connection header received
0924 16:12:15.275238 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:15.275245 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:15.275496 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:15.275670 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:12:15.276637 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.278287 (+ 1650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.280884 (+ 2597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.282346 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.282466 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:15.282676 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:15.283824 (+ 1148us) server_negotiation.cc:299] Negotiation successful
0924 16:12:15.284103 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10797,"thread_start_us":10617,"threads_started":1}
I20260924 16:12:15.287626 32059 tablet_service.cc:1467] Processing CreateTablet for tablet d32a936f850a4a4aa4679ae31a2376e9 (DEFAULT_TABLE table=test-table [id=95401dfe2b3d488d93913e3f53105edf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:12:15.287768 32206 tablet_service.cc:1467] Processing CreateTablet for tablet d32a936f850a4a4aa4679ae31a2376e9 (DEFAULT_TABLE table=test-table [id=95401dfe2b3d488d93913e3f53105edf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:12:15.287640 31918 tablet_service.cc:1467] Processing CreateTablet for tablet d32a936f850a4a4aa4679ae31a2376e9 (DEFAULT_TABLE table=test-table [id=95401dfe2b3d488d93913e3f53105edf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:12:15.288937 32059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d32a936f850a4a4aa4679ae31a2376e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:15.289059 32206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d32a936f850a4a4aa4679ae31a2376e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:15.289049 31918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d32a936f850a4a4aa4679ae31a2376e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:15.301563 32295 tablet_bootstrap.cc:492] T d32a936f850a4a4aa4679ae31a2376e9 P fb09133e8ecc409ba9ffc6044c90180f: Bootstrap starting.
I20260924 16:12:15.306090 32295 tablet_bootstrap.cc:654] T d32a936f850a4a4aa4679ae31a2376e9 P fb09133e8ecc409ba9ffc6044c90180f: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:15.307794 32295 log.cc:824] T d32a936f850a4a4aa4679ae31a2376e9 P fb09133e8ecc409ba9ffc6044c90180f: Log is configured to *not* fsync() on all Append() calls
I20260924 16:12:15.308930 32296 tablet_bootstrap.cc:492] T d32a936f850a4a4aa4679ae31a2376e9 P f81b244edb0c46c38601587481b42ef0: Bootstrap starting.
I20260924 16:12:15.312480 32295 tablet_bootstrap.cc:492] T d32a936f850a4a4aa4679ae31a2376e9 P fb09133e8ecc409ba9ffc6044c90180f: No bootstrap required, opened a new log
I20260924 16:12:15.312855 32295 ts_tablet_manager.cc:1397] T d32a936f850a4a4aa4679ae31a2376e9 P fb09133e8ecc409ba9ffc6044c90180f: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20260924 16:12:15.313263 32297 tablet_bootstrap.cc:492] T d32a936f850a4a4aa4679ae31a2376e9 P ba716e8ff56743de8c3094b708c833a7: Bootstrap starting.
I20260924 16:12:15.314643 32296 tablet_bootstrap.cc:654] T d32a936f850a4a4aa4679ae31a2376e9 P f81b244edb0c46c38601587481b42ef0: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:15.315795 32296 log.cc:824] T d32a936f850a4a4aa4679ae31a2376e9 P f81b244edb0c46c38601587481b42ef0: Log is configured to *not* fsync() on all Append() calls
I20260924 16:12:15.317515 32297 tablet_bootstrap.cc:654] T d32a936f850a4a4aa4679ae31a2376e9 P ba716e8ff56743de8c3094b708c833a7: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:15.318733 32297 log.cc:824] T d32a936f850a4a4aa4679ae31a2376e9 P ba716e8ff56743de8c3094b708c833a7: Log is configured to *not* fsync() on all Append() calls
I20260924 16:12:15.319943 32296 tablet_bootstrap.cc:492] T d32a936f850a4a4aa4679ae31a2376e9 P f81b244edb0c46c38601587481b42ef0: No bootstrap required, opened a new log
I20260924 16:12:15.320240 32296 ts_tablet_manager.cc:1397] T d32a936f850a4a4aa4679ae31a2376e9 P f81b244edb0c46c38601587481b42ef0: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260924 16:12:15.321823 32297 tablet_bootstrap.cc:492] T d32a936f850a4a4aa4679ae31a2376e9 P ba716e8ff56743de8c3094b708c833a7: No bootstrap required, opened a new log
I20260924 16:12:15.322120 32297 ts_tablet_manager.cc:1397] T d32a936f850a4a4aa4679ae31a2376e9 P ba716e8ff56743de8c3094b708c833a7: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20260924 16:12:15.322705 32295 raft_consensus.cc:348] T d32a936f850a4a4aa4679ae31a2376e9 P fb09133e8ecc409ba9ffc6044c90180f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f81b244edb0c46c38601587481b42ef0" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 35217 } } peers { permanent_uuid: "ba716e8ff56743de8c3094b708c833a7" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 39383 } } peers { permanent_uuid: "fb09133e8ecc409ba9ffc6044c90180f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 44825 } }
I20260924 16:12:15.323112 32295 raft_consensus.cc:374] T d32a936f850a4a4aa4679ae31a2376e9 P fb09133e8ecc409ba9ffc6044c90180f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:15.323237 32295 raft_consensus.cc:729] T d32a936f850a4a4aa4679ae31a2376e9 P fb09133e8ecc409ba9ffc6044c90180f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb09133e8ecc409ba9ffc6044c90180f, State: Initialized, Role: FOLLOWER
I20260924 16:12:15.323891 32295 consensus_queue.cc:260] T d32a936f850a4a4aa4679ae31a2376e9 P fb09133e8ecc409ba9ffc6044c90180f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f81b244edb0c46c38601587481b42ef0" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 35217 } } peers { permanent_uuid: "ba716e8ff56743de8c3094b708c833a7" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 39383 } } peers { permanent_uuid: "fb09133e8ecc409ba9ffc6044c90180f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 44825 } }
I20260924 16:12:15.326802 32131 heartbeater.cc:503] Master 127.29.234.62:39697 was elected leader, sending a full tablet report...
I20260924 16:12:15.327363 32295 ts_tablet_manager.cc:1428] T d32a936f850a4a4aa4679ae31a2376e9 P fb09133e8ecc409ba9ffc6044c90180f: Time spent starting tablet: real 0.014s user 0.008s sys 0.006s
I20260924 16:12:15.333104 32297 raft_consensus.cc:348] T d32a936f850a4a4aa4679ae31a2376e9 P ba716e8ff56743de8c3094b708c833a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f81b244edb0c46c38601587481b42ef0" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 35217 } } peers { permanent_uuid: "ba716e8ff56743de8c3094b708c833a7" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 39383 } } peers { permanent_uuid: "fb09133e8ecc409ba9ffc6044c90180f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 44825 } }
I20260924 16:12:15.333103 32296 raft_consensus.cc:348] T d32a936f850a4a4aa4679ae31a2376e9 P f81b244edb0c46c38601587481b42ef0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f81b244edb0c46c38601587481b42ef0" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 35217 } } peers { permanent_uuid: "ba716e8ff56743de8c3094b708c833a7" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 39383 } } peers { permanent_uuid: "fb09133e8ecc409ba9ffc6044c90180f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 44825 } }
I20260924 16:12:15.333652 32297 raft_consensus.cc:374] T d32a936f850a4a4aa4679ae31a2376e9 P ba716e8ff56743de8c3094b708c833a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:15.333652 32296 raft_consensus.cc:374] T d32a936f850a4a4aa4679ae31a2376e9 P f81b244edb0c46c38601587481b42ef0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:15.333763 32296 raft_consensus.cc:729] T d32a936f850a4a4aa4679ae31a2376e9 P f81b244edb0c46c38601587481b42ef0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f81b244edb0c46c38601587481b42ef0, State: Initialized, Role: FOLLOWER
I20260924 16:12:15.333774 32297 raft_consensus.cc:729] T d32a936f850a4a4aa4679ae31a2376e9 P ba716e8ff56743de8c3094b708c833a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba716e8ff56743de8c3094b708c833a7, State: Initialized, Role: FOLLOWER
I20260924 16:12:15.334474 32296 consensus_queue.cc:260] T d32a936f850a4a4aa4679ae31a2376e9 P f81b244edb0c46c38601587481b42ef0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f81b244edb0c46c38601587481b42ef0" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 35217 } } peers { permanent_uuid: "ba716e8ff56743de8c3094b708c833a7" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 39383 } } peers { permanent_uuid: "fb09133e8ecc409ba9ffc6044c90180f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 44825 } }
I20260924 16:12:15.334474 32297 consensus_queue.cc:260] T d32a936f850a4a4aa4679ae31a2376e9 P ba716e8ff56743de8c3094b708c833a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f81b244edb0c46c38601587481b42ef0" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 35217 } } peers { permanent_uuid: "ba716e8ff56743de8c3094b708c833a7" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 39383 } } peers { permanent_uuid: "fb09133e8ecc409ba9ffc6044c90180f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 44825 } }
I20260924 16:12:15.336107 32273 heartbeater.cc:503] Master 127.29.234.62:39697 was elected leader, sending a full tablet report...
I20260924 16:12:15.336762 32297 ts_tablet_manager.cc:1428] T d32a936f850a4a4aa4679ae31a2376e9 P ba716e8ff56743de8c3094b708c833a7: Time spent starting tablet: real 0.015s user 0.013s sys 0.001s
I20260924 16:12:15.337728 32296 ts_tablet_manager.cc:1428] T d32a936f850a4a4aa4679ae31a2376e9 P f81b244edb0c46c38601587481b42ef0: Time spent starting tablet: real 0.017s user 0.017s sys 0.000s
W20260924 16:12:15.387148 31990 tablet.cc:2369] T d32a936f850a4a4aa4679ae31a2376e9 P f81b244edb0c46c38601587481b42ef0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 16:12:15.397967 32132 tablet.cc:2369] T d32a936f850a4a4aa4679ae31a2376e9 P fb09133e8ecc409ba9ffc6044c90180f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 16:12:15.432651 32274 tablet.cc:2369] T d32a936f850a4a4aa4679ae31a2376e9 P ba716e8ff56743de8c3094b708c833a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:12:15.559731 32301 raft_consensus.cc:482] T d32a936f850a4a4aa4679ae31a2376e9 P fb09133e8ecc409ba9ffc6044c90180f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:12:15.560055 32301 raft_consensus.cc:504] T d32a936f850a4a4aa4679ae31a2376e9 P fb09133e8ecc409ba9ffc6044c90180f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f81b244edb0c46c38601587481b42ef0" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 35217 } } peers { permanent_uuid: "ba716e8ff56743de8c3094b708c833a7" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 39383 } } peers { permanent_uuid: "fb09133e8ecc409ba9ffc6044c90180f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 44825 } }
I20260924 16:12:15.561887 32301 leader_election.cc:288] T d32a936f850a4a4aa4679ae31a2376e9 P fb09133e8ecc409ba9ffc6044c90180f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f81b244edb0c46c38601587481b42ef0 (127.29.234.1:35217), ba716e8ff56743de8c3094b708c833a7 (127.29.234.3:39383)
I20260924 16:12:15.570452 32288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:15.562201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.2:42673 (local address 127.29.234.1:35217)
0924 16:12:15.562458 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:15.562467 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:15.562930 (+ 463us) server_negotiation.cc:407] Connection header received
0924 16:12:15.563152 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:15.563156 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:15.563241 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:15.563373 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:12:15.564130 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.565274 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.567769 (+ 2495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.569045 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.569199 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:15.569281 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:15.570057 (+ 776us) server_negotiation.cc:299] Negotiation successful
0924 16:12:15.570191 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 16:12:15.570452 32291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:15.562444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.2:48441 (local address 127.29.234.3:39383)
0924 16:12:15.562648 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:15.562657 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:15.563246 (+ 589us) server_negotiation.cc:407] Connection header received
0924 16:12:15.563352 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:15.563358 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:15.563431 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:15.563519 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:12:15.564231 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.565274 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.567858 (+ 2584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.569045 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.569199 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:15.569273 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:15.570057 (+ 784us) server_negotiation.cc:299] Negotiation successful
0924 16:12:15.570191 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 16:12:15.570722 32305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:15.562357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.3:39383 (local address 127.29.234.2:48441)
0924 16:12:15.563154 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:15.563167 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:15.563251 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:15.563557 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:15.563567 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:15.563599 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:12:15.564058 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:15.564072 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.565415 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.565418 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:15.567704 (+ 2286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:15.567720 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.569524 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.569531 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:15.569817 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:15.569911 (+ 94us) client_negotiation.cc:769] Sending connection context
0924 16:12:15.570009 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 16:12:15.570200 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"mutex_wait_us":198,"thread_start_us":431,"threads_started":1}
I20260924 16:12:15.570747 32304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:15.562068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:35217 (local address 127.29.234.2:42673)
0924 16:12:15.562780 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:15.562803 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:15.562996 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:15.563398 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:15.563402 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:15.563420 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:12:15.563964 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:15.563984 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.565413 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.565418 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:15.567603 (+ 2185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:15.567619 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.569534 (+ 1915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.569541 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:15.569822 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:15.569909 (+ 87us) client_negotiation.cc:769] Sending connection context
0924 16:12:15.570008 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 16:12:15.570199 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"mutex_wait_us":201,"spinlock_wait_cycles":24448,"thread_start_us":406,"threads_started":1}
I20260924 16:12:15.571807 31944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d32a936f850a4a4aa4679ae31a2376e9" candidate_uuid: "fb09133e8ecc409ba9ffc6044c90180f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f81b244edb0c46c38601587481b42ef0" is_pre_election: true
I20260924 16:12:15.571991 32226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d32a936f850a4a4aa4679ae31a2376e9" candidate_uuid: "fb09133e8ecc409ba9ffc6044c90180f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba716e8ff56743de8c3094b708c833a7" is_pre_election: true
I20260924 16:12:15.572427 31944 raft_consensus.cc:2454] T d32a936f850a4a4aa4679ae31a2376e9 P f81b244edb0c46c38601587481b42ef0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb09133e8ecc409ba9ffc6044c90180f in term 0.
I20260924 16:12:15.572664 32226 raft_consensus.cc:2454] T d32a936f850a4a4aa4679ae31a2376e9 P ba716e8ff56743de8c3094b708c833a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb09133e8ecc409ba9ffc6044c90180f in term 0.
I20260924 16:12:15.573217 32012 leader_election.cc:302] T d32a936f850a4a4aa4679ae31a2376e9 P fb09133e8ecc409ba9ffc6044c90180f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f81b244edb0c46c38601587481b42ef0, fb09133e8ecc409ba9ffc6044c90180f; no voters:
I20260924 16:12:15.575544 32301 raft_consensus.cc:2781] T d32a936f850a4a4aa4679ae31a2376e9 P fb09133e8ecc409ba9ffc6044c90180f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:12:15.575739 32301 raft_consensus.cc:482] T d32a936f850a4a4aa4679ae31a2376e9 P fb09133e8ecc409ba9ffc6044c90180f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:12:15.575863 32301 raft_consensus.cc:3037] T d32a936f850a4a4aa4679ae31a2376e9 P fb09133e8ecc409ba9ffc6044c90180f [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:15.579187 32301 raft_consensus.cc:504] T d32a936f850a4a4aa4679ae31a2376e9 P fb09133e8ecc409ba9ffc6044c90180f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f81b244edb0c46c38601587481b42ef0" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 35217 } } peers { permanent_uuid: "ba716e8ff56743de8c3094b708c833a7" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 39383 } } peers { permanent_uuid: "fb09133e8ecc409ba9ffc6044c90180f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 44825 } }
I20260924 16:12:15.579867 32301 leader_election.cc:288] T d32a936f850a4a4aa4679ae31a2376e9 P fb09133e8ecc409ba9ffc6044c90180f [CANDIDATE]: Term 1 election: Requested vote from peers f81b244edb0c46c38601587481b42ef0 (127.29.234.1:35217), ba716e8ff56743de8c3094b708c833a7 (127.29.234.3:39383)
I20260924 16:12:15.580437 31944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d32a936f850a4a4aa4679ae31a2376e9" candidate_uuid: "fb09133e8ecc409ba9ffc6044c90180f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f81b244edb0c46c38601587481b42ef0"
I20260924 16:12:15.580467 32226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d32a936f850a4a4aa4679ae31a2376e9" candidate_uuid: "fb09133e8ecc409ba9ffc6044c90180f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba716e8ff56743de8c3094b708c833a7"
I20260924 16:12:15.580711 32226 raft_consensus.cc:3037] T d32a936f850a4a4aa4679ae31a2376e9 P ba716e8ff56743de8c3094b708c833a7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:15.580718 31944 raft_consensus.cc:3037] T d32a936f850a4a4aa4679ae31a2376e9 P f81b244edb0c46c38601587481b42ef0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:15.583281 32226 raft_consensus.cc:2454] T d32a936f850a4a4aa4679ae31a2376e9 P ba716e8ff56743de8c3094b708c833a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb09133e8ecc409ba9ffc6044c90180f in term 1.
I20260924 16:12:15.583431 31944 raft_consensus.cc:2454] T d32a936f850a4a4aa4679ae31a2376e9 P f81b244edb0c46c38601587481b42ef0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb09133e8ecc409ba9ffc6044c90180f in term 1.
I20260924 16:12:15.583766 32013 leader_election.cc:302] T d32a936f850a4a4aa4679ae31a2376e9 P fb09133e8ecc409ba9ffc6044c90180f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba716e8ff56743de8c3094b708c833a7, fb09133e8ecc409ba9ffc6044c90180f; no voters:
I20260924 16:12:15.584076 32301 raft_consensus.cc:2781] T d32a936f850a4a4aa4679ae31a2376e9 P fb09133e8ecc409ba9ffc6044c90180f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:12:15.584499 32301 raft_consensus.cc:686] T d32a936f850a4a4aa4679ae31a2376e9 P fb09133e8ecc409ba9ffc6044c90180f [term 1 LEADER]: Becoming Leader. State: Replica: fb09133e8ecc409ba9ffc6044c90180f, State: Running, Role: LEADER
I20260924 16:12:15.585067 32301 consensus_queue.cc:237] T d32a936f850a4a4aa4679ae31a2376e9 P fb09133e8ecc409ba9ffc6044c90180f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f81b244edb0c46c38601587481b42ef0" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 35217 } } peers { permanent_uuid: "ba716e8ff56743de8c3094b708c833a7" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 39383 } } peers { permanent_uuid: "fb09133e8ecc409ba9ffc6044c90180f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 44825 } }
I20260924 16:12:15.589742 31779 catalog_manager.cc:5495] T d32a936f850a4a4aa4679ae31a2376e9 P fb09133e8ecc409ba9ffc6044c90180f reported cstate change: term changed from 0 to 1, leader changed from <none> to fb09133e8ecc409ba9ffc6044c90180f (127.29.234.2). New cstate: current_term: 1 leader_uuid: "fb09133e8ecc409ba9ffc6044c90180f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f81b244edb0c46c38601587481b42ef0" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 35217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ba716e8ff56743de8c3094b708c833a7" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 39383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb09133e8ecc409ba9ffc6044c90180f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 44825 } health_report { overall_health: HEALTHY } } }
I20260924 16:12:15.661623 32294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:15.655783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50388 (local address 127.29.234.2:44825)
0924 16:12:15.656045 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:15.656050 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:15.656068 (+ 18us) server_negotiation.cc:407] Connection header received
0924 16:12:15.656404 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:15.656408 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:15.656504 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:15.656621 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:12:15.657096 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.658163 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.659516 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.659939 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.660049 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:15.660453 (+ 404us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:12:15.661000 (+ 547us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:12:15.661134 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:15.661263 (+ 129us) server_negotiation.cc:299] Negotiation successful
0924 16:12:15.661399 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260924 16:12:15.669859 32226 raft_consensus.cc:1261] T d32a936f850a4a4aa4679ae31a2376e9 P ba716e8ff56743de8c3094b708c833a7 [term 1 FOLLOWER]: Refusing update from remote peer fb09133e8ecc409ba9ffc6044c90180f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:12:15.669922 31944 raft_consensus.cc:1261] T d32a936f850a4a4aa4679ae31a2376e9 P f81b244edb0c46c38601587481b42ef0 [term 1 FOLLOWER]: Refusing update from remote peer fb09133e8ecc409ba9ffc6044c90180f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:12:15.670857 32306 consensus_queue.cc:1035] T d32a936f850a4a4aa4679ae31a2376e9 P fb09133e8ecc409ba9ffc6044c90180f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f81b244edb0c46c38601587481b42ef0" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 35217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:12:15.671232 32301 consensus_queue.cc:1035] T d32a936f850a4a4aa4679ae31a2376e9 P fb09133e8ecc409ba9ffc6044c90180f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba716e8ff56743de8c3094b708c833a7" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 39383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:12:15.703876 31779 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50730:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:12:15.704181 31779 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50730:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:12:15.708134 31779 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dde4f26be06a4b1799e168313c156528: Sending DeleteTablet for 3 replicas of tablet d32a936f850a4a4aa4679ae31a2376e9
I20260924 16:12:15.709438 32206 tablet_service.cc:1514] Processing DeleteTablet for tablet d32a936f850a4a4aa4679ae31a2376e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:12:15 UTC) from {username='kudu', principal='kudu/127.29.234.62@KRBTEST.COM'} at 127.0.0.1:42636
I20260924 16:12:15.709435 31918 tablet_service.cc:1514] Processing DeleteTablet for tablet d32a936f850a4a4aa4679ae31a2376e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:12:15 UTC) from {username='kudu', principal='kudu/127.29.234.62@KRBTEST.COM'} at 127.0.0.1:58496
I20260924 16:12:15.709513 32059 tablet_service.cc:1514] Processing DeleteTablet for tablet d32a936f850a4a4aa4679ae31a2376e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:12:15 UTC) from {username='kudu', principal='kudu/127.29.234.62@KRBTEST.COM'} at 127.0.0.1:50374
I20260924 16:12:15.713796 31779 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50730:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:12:15.714545 31779 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:12:15.722268 32320 tablet_replica.cc:330] T d32a936f850a4a4aa4679ae31a2376e9 P f81b244edb0c46c38601587481b42ef0: stopping tablet replica
I20260924 16:12:15.723134 32320 raft_consensus.cc:2229] T d32a936f850a4a4aa4679ae31a2376e9 P f81b244edb0c46c38601587481b42ef0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:12:15.723927 32320 raft_consensus.cc:2258] T d32a936f850a4a4aa4679ae31a2376e9 P f81b244edb0c46c38601587481b42ef0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:12:15.724092 32205 tablet_service.cc:1467] Processing CreateTablet for tablet dfb42459c86140f8bd9cd1b798e5a197 (DEFAULT_TABLE table=test-table [id=37c4b5c5ec534d22997f524fd6ca8d93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:12:15.724592 32205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfb42459c86140f8bd9cd1b798e5a197. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:15.727002 32320 ts_tablet_manager.cc:1905] T d32a936f850a4a4aa4679ae31a2376e9 P f81b244edb0c46c38601587481b42ef0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:12:15.727725 32319 tablet_replica.cc:330] T d32a936f850a4a4aa4679ae31a2376e9 P ba716e8ff56743de8c3094b708c833a7: stopping tablet replica
I20260924 16:12:15.728606 32058 tablet_service.cc:1467] Processing CreateTablet for tablet dfb42459c86140f8bd9cd1b798e5a197 (DEFAULT_TABLE table=test-table [id=37c4b5c5ec534d22997f524fd6ca8d93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:12:15.728818 32319 raft_consensus.cc:2229] T d32a936f850a4a4aa4679ae31a2376e9 P ba716e8ff56743de8c3094b708c833a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:12:15.729514 32319 raft_consensus.cc:2258] T d32a936f850a4a4aa4679ae31a2376e9 P ba716e8ff56743de8c3094b708c833a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:12:15.728521 31918 tablet_service.cc:1467] Processing CreateTablet for tablet dfb42459c86140f8bd9cd1b798e5a197 (DEFAULT_TABLE table=test-table [id=37c4b5c5ec534d22997f524fd6ca8d93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:12:15.729737 32058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfb42459c86140f8bd9cd1b798e5a197. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:15.730042 31918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfb42459c86140f8bd9cd1b798e5a197. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:15.738358 32296 tablet_bootstrap.cc:492] T dfb42459c86140f8bd9cd1b798e5a197 P f81b244edb0c46c38601587481b42ef0: Bootstrap starting.
I20260924 16:12:15.738412 32321 tablet_replica.cc:330] T d32a936f850a4a4aa4679ae31a2376e9 P fb09133e8ecc409ba9ffc6044c90180f: stopping tablet replica
I20260924 16:12:15.738873 32321 raft_consensus.cc:2229] T d32a936f850a4a4aa4679ae31a2376e9 P fb09133e8ecc409ba9ffc6044c90180f [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:12:15.739666 32321 raft_consensus.cc:2258] T d32a936f850a4a4aa4679ae31a2376e9 P fb09133e8ecc409ba9ffc6044c90180f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:12:15.740136 32296 tablet_bootstrap.cc:654] T dfb42459c86140f8bd9cd1b798e5a197 P f81b244edb0c46c38601587481b42ef0: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:15.742264 32319 ts_tablet_manager.cc:1905] T d32a936f850a4a4aa4679ae31a2376e9 P ba716e8ff56743de8c3094b708c833a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:12:15.746497 32295 tablet_bootstrap.cc:492] T dfb42459c86140f8bd9cd1b798e5a197 P fb09133e8ecc409ba9ffc6044c90180f: Bootstrap starting.
I20260924 16:12:15.748103 32321 ts_tablet_manager.cc:1905] T d32a936f850a4a4aa4679ae31a2376e9 P fb09133e8ecc409ba9ffc6044c90180f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:12:15.746340 32297 tablet_bootstrap.cc:492] T dfb42459c86140f8bd9cd1b798e5a197 P ba716e8ff56743de8c3094b708c833a7: Bootstrap starting.
I20260924 16:12:15.749939 32297 tablet_bootstrap.cc:654] T dfb42459c86140f8bd9cd1b798e5a197 P ba716e8ff56743de8c3094b708c833a7: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:15.750797 32320 ts_tablet_manager.cc:1918] T d32a936f850a4a4aa4679ae31a2376e9 P f81b244edb0c46c38601587481b42ef0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:12:15.750999 32320 log.cc:1196] T d32a936f850a4a4aa4679ae31a2376e9 P f81b244edb0c46c38601587481b42ef0: Deleting WAL directory at /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/minicluster-data/ts-0/wal/wals/d32a936f850a4a4aa4679ae31a2376e9
I20260924 16:12:15.751719 32320 ts_tablet_manager.cc:1939] T d32a936f850a4a4aa4679ae31a2376e9 P f81b244edb0c46c38601587481b42ef0: Deleting consensus metadata
I20260924 16:12:15.754050 31765 catalog_manager.cc:4841] TS f81b244edb0c46c38601587481b42ef0 (127.29.234.1:35217): tablet d32a936f850a4a4aa4679ae31a2376e9 (table test-table [id=95401dfe2b3d488d93913e3f53105edf]) successfully deleted
I20260924 16:12:15.748289 32295 tablet_bootstrap.cc:654] T dfb42459c86140f8bd9cd1b798e5a197 P fb09133e8ecc409ba9ffc6044c90180f: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:15.755286 32297 tablet_bootstrap.cc:492] T dfb42459c86140f8bd9cd1b798e5a197 P ba716e8ff56743de8c3094b708c833a7: No bootstrap required, opened a new log
I20260924 16:12:15.755527 32297 ts_tablet_manager.cc:1397] T dfb42459c86140f8bd9cd1b798e5a197 P ba716e8ff56743de8c3094b708c833a7: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260924 16:12:15.756798 32297 raft_consensus.cc:348] T dfb42459c86140f8bd9cd1b798e5a197 P ba716e8ff56743de8c3094b708c833a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f81b244edb0c46c38601587481b42ef0" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 35217 } } peers { permanent_uuid: "fb09133e8ecc409ba9ffc6044c90180f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 44825 } } peers { permanent_uuid: "ba716e8ff56743de8c3094b708c833a7" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 39383 } }
I20260924 16:12:15.757396 32297 raft_consensus.cc:374] T dfb42459c86140f8bd9cd1b798e5a197 P ba716e8ff56743de8c3094b708c833a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:15.757593 32297 raft_consensus.cc:729] T dfb42459c86140f8bd9cd1b798e5a197 P ba716e8ff56743de8c3094b708c833a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba716e8ff56743de8c3094b708c833a7, State: Initialized, Role: FOLLOWER
I20260924 16:12:15.758144 32297 consensus_queue.cc:260] T dfb42459c86140f8bd9cd1b798e5a197 P ba716e8ff56743de8c3094b708c833a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f81b244edb0c46c38601587481b42ef0" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 35217 } } peers { permanent_uuid: "fb09133e8ecc409ba9ffc6044c90180f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 44825 } } peers { permanent_uuid: "ba716e8ff56743de8c3094b708c833a7" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 39383 } }
I20260924 16:12:15.760212 32297 ts_tablet_manager.cc:1428] T dfb42459c86140f8bd9cd1b798e5a197 P ba716e8ff56743de8c3094b708c833a7: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260924 16:12:15.767453 32296 tablet_bootstrap.cc:492] T dfb42459c86140f8bd9cd1b798e5a197 P f81b244edb0c46c38601587481b42ef0: No bootstrap required, opened a new log
I20260924 16:12:15.767660 32295 tablet_bootstrap.cc:492] T dfb42459c86140f8bd9cd1b798e5a197 P fb09133e8ecc409ba9ffc6044c90180f: No bootstrap required, opened a new log
I20260924 16:12:15.767786 32296 ts_tablet_manager.cc:1397] T dfb42459c86140f8bd9cd1b798e5a197 P f81b244edb0c46c38601587481b42ef0: Time spent bootstrapping tablet: real 0.029s user 0.010s sys 0.005s
I20260924 16:12:15.768118 32295 ts_tablet_manager.cc:1397] T dfb42459c86140f8bd9cd1b798e5a197 P fb09133e8ecc409ba9ffc6044c90180f: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.007s
I20260924 16:12:15.768457 32321 ts_tablet_manager.cc:1918] T d32a936f850a4a4aa4679ae31a2376e9 P fb09133e8ecc409ba9ffc6044c90180f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:12:15.768635 32321 log.cc:1196] T d32a936f850a4a4aa4679ae31a2376e9 P fb09133e8ecc409ba9ffc6044c90180f: Deleting WAL directory at /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/minicluster-data/ts-1/wal/wals/d32a936f850a4a4aa4679ae31a2376e9
I20260924 16:12:15.769057 32295 raft_consensus.cc:348] T dfb42459c86140f8bd9cd1b798e5a197 P fb09133e8ecc409ba9ffc6044c90180f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f81b244edb0c46c38601587481b42ef0" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 35217 } } peers { permanent_uuid: "fb09133e8ecc409ba9ffc6044c90180f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 44825 } } peers { permanent_uuid: "ba716e8ff56743de8c3094b708c833a7" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 39383 } }
I20260924 16:12:15.769366 32295 raft_consensus.cc:374] T dfb42459c86140f8bd9cd1b798e5a197 P fb09133e8ecc409ba9ffc6044c90180f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:15.769486 32295 raft_consensus.cc:729] T dfb42459c86140f8bd9cd1b798e5a197 P fb09133e8ecc409ba9ffc6044c90180f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb09133e8ecc409ba9ffc6044c90180f, State: Initialized, Role: FOLLOWER
I20260924 16:12:15.769768 32295 consensus_queue.cc:260] T dfb42459c86140f8bd9cd1b798e5a197 P fb09133e8ecc409ba9ffc6044c90180f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f81b244edb0c46c38601587481b42ef0" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 35217 } } peers { permanent_uuid: "fb09133e8ecc409ba9ffc6044c90180f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 44825 } } peers { permanent_uuid: "ba716e8ff56743de8c3094b708c833a7" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 39383 } }
I20260924 16:12:15.770012 32296 raft_consensus.cc:348] T dfb42459c86140f8bd9cd1b798e5a197 P f81b244edb0c46c38601587481b42ef0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f81b244edb0c46c38601587481b42ef0" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 35217 } } peers { permanent_uuid: "fb09133e8ecc409ba9ffc6044c90180f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 44825 } } peers { permanent_uuid: "ba716e8ff56743de8c3094b708c833a7" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 39383 } }
I20260924 16:12:15.770280 32296 raft_consensus.cc:374] T dfb42459c86140f8bd9cd1b798e5a197 P f81b244edb0c46c38601587481b42ef0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:15.770381 32296 raft_consensus.cc:729] T dfb42459c86140f8bd9cd1b798e5a197 P f81b244edb0c46c38601587481b42ef0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f81b244edb0c46c38601587481b42ef0, State: Initialized, Role: FOLLOWER
I20260924 16:12:15.770561 32295 ts_tablet_manager.cc:1428] T dfb42459c86140f8bd9cd1b798e5a197 P fb09133e8ecc409ba9ffc6044c90180f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 16:12:15.769215 32321 ts_tablet_manager.cc:1939] T d32a936f850a4a4aa4679ae31a2376e9 P fb09133e8ecc409ba9ffc6044c90180f: Deleting consensus metadata
I20260924 16:12:15.770641 32296 consensus_queue.cc:260] T dfb42459c86140f8bd9cd1b798e5a197 P f81b244edb0c46c38601587481b42ef0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f81b244edb0c46c38601587481b42ef0" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 35217 } } peers { permanent_uuid: "fb09133e8ecc409ba9ffc6044c90180f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 44825 } } peers { permanent_uuid: "ba716e8ff56743de8c3094b708c833a7" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 39383 } }
I20260924 16:12:15.771287 32296 ts_tablet_manager.cc:1428] T dfb42459c86140f8bd9cd1b798e5a197 P f81b244edb0c46c38601587481b42ef0: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 16:12:15.772569 31764 catalog_manager.cc:4841] TS fb09133e8ecc409ba9ffc6044c90180f (127.29.234.2:44825): tablet d32a936f850a4a4aa4679ae31a2376e9 (table test-table [id=95401dfe2b3d488d93913e3f53105edf]) successfully deleted
I20260924 16:12:15.773079 32319 ts_tablet_manager.cc:1918] T d32a936f850a4a4aa4679ae31a2376e9 P ba716e8ff56743de8c3094b708c833a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:12:15.773300 32319 log.cc:1196] T d32a936f850a4a4aa4679ae31a2376e9 P ba716e8ff56743de8c3094b708c833a7: Deleting WAL directory at /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/minicluster-data/ts-2/wal/wals/d32a936f850a4a4aa4679ae31a2376e9
I20260924 16:12:15.773922 32319 ts_tablet_manager.cc:1939] T d32a936f850a4a4aa4679ae31a2376e9 P ba716e8ff56743de8c3094b708c833a7: Deleting consensus metadata
I20260924 16:12:15.775856 31766 catalog_manager.cc:4841] TS ba716e8ff56743de8c3094b708c833a7 (127.29.234.3:39383): tablet d32a936f850a4a4aa4679ae31a2376e9 (table test-table [id=95401dfe2b3d488d93913e3f53105edf]) successfully deleted
I20260924 16:12:15.835561 32314 raft_consensus.cc:482] T dfb42459c86140f8bd9cd1b798e5a197 P fb09133e8ecc409ba9ffc6044c90180f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:12:15.835813 32314 raft_consensus.cc:504] T dfb42459c86140f8bd9cd1b798e5a197 P fb09133e8ecc409ba9ffc6044c90180f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f81b244edb0c46c38601587481b42ef0" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 35217 } } peers { permanent_uuid: "fb09133e8ecc409ba9ffc6044c90180f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 44825 } } peers { permanent_uuid: "ba716e8ff56743de8c3094b708c833a7" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 39383 } }
I20260924 16:12:15.836772 32314 leader_election.cc:288] T dfb42459c86140f8bd9cd1b798e5a197 P fb09133e8ecc409ba9ffc6044c90180f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f81b244edb0c46c38601587481b42ef0 (127.29.234.1:35217), ba716e8ff56743de8c3094b708c833a7 (127.29.234.3:39383)
I20260924 16:12:15.837682 32226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfb42459c86140f8bd9cd1b798e5a197" candidate_uuid: "fb09133e8ecc409ba9ffc6044c90180f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba716e8ff56743de8c3094b708c833a7" is_pre_election: true
I20260924 16:12:15.838023 31944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfb42459c86140f8bd9cd1b798e5a197" candidate_uuid: "fb09133e8ecc409ba9ffc6044c90180f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f81b244edb0c46c38601587481b42ef0" is_pre_election: true
I20260924 16:12:15.838308 31944 raft_consensus.cc:2454] T dfb42459c86140f8bd9cd1b798e5a197 P f81b244edb0c46c38601587481b42ef0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb09133e8ecc409ba9ffc6044c90180f in term 0.
I20260924 16:12:15.838605 32226 raft_consensus.cc:2454] T dfb42459c86140f8bd9cd1b798e5a197 P ba716e8ff56743de8c3094b708c833a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb09133e8ecc409ba9ffc6044c90180f in term 0.
I20260924 16:12:15.838960 32012 leader_election.cc:302] T dfb42459c86140f8bd9cd1b798e5a197 P fb09133e8ecc409ba9ffc6044c90180f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f81b244edb0c46c38601587481b42ef0, fb09133e8ecc409ba9ffc6044c90180f; no voters:
I20260924 16:12:15.839452 32314 raft_consensus.cc:2781] T dfb42459c86140f8bd9cd1b798e5a197 P fb09133e8ecc409ba9ffc6044c90180f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:12:15.839829 32314 raft_consensus.cc:482] T dfb42459c86140f8bd9cd1b798e5a197 P fb09133e8ecc409ba9ffc6044c90180f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:12:15.840085 32314 raft_consensus.cc:3037] T dfb42459c86140f8bd9cd1b798e5a197 P fb09133e8ecc409ba9ffc6044c90180f [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:15.844499 32314 raft_consensus.cc:504] T dfb42459c86140f8bd9cd1b798e5a197 P fb09133e8ecc409ba9ffc6044c90180f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f81b244edb0c46c38601587481b42ef0" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 35217 } } peers { permanent_uuid: "fb09133e8ecc409ba9ffc6044c90180f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 44825 } } peers { permanent_uuid: "ba716e8ff56743de8c3094b708c833a7" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 39383 } }
I20260924 16:12:15.845470 32314 leader_election.cc:288] T dfb42459c86140f8bd9cd1b798e5a197 P fb09133e8ecc409ba9ffc6044c90180f [CANDIDATE]: Term 1 election: Requested vote from peers f81b244edb0c46c38601587481b42ef0 (127.29.234.1:35217), ba716e8ff56743de8c3094b708c833a7 (127.29.234.3:39383)
I20260924 16:12:15.846098 31944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfb42459c86140f8bd9cd1b798e5a197" candidate_uuid: "fb09133e8ecc409ba9ffc6044c90180f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f81b244edb0c46c38601587481b42ef0"
I20260924 16:12:15.846345 31944 raft_consensus.cc:3037] T dfb42459c86140f8bd9cd1b798e5a197 P f81b244edb0c46c38601587481b42ef0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:15.846625 32226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfb42459c86140f8bd9cd1b798e5a197" candidate_uuid: "fb09133e8ecc409ba9ffc6044c90180f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba716e8ff56743de8c3094b708c833a7"
I20260924 16:12:15.846895 32226 raft_consensus.cc:3037] T dfb42459c86140f8bd9cd1b798e5a197 P ba716e8ff56743de8c3094b708c833a7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:15.849028 32226 raft_consensus.cc:2454] T dfb42459c86140f8bd9cd1b798e5a197 P ba716e8ff56743de8c3094b708c833a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb09133e8ecc409ba9ffc6044c90180f in term 1.
I20260924 16:12:15.849321 31944 raft_consensus.cc:2454] T dfb42459c86140f8bd9cd1b798e5a197 P f81b244edb0c46c38601587481b42ef0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb09133e8ecc409ba9ffc6044c90180f in term 1.
I20260924 16:12:15.849488 32013 leader_election.cc:302] T dfb42459c86140f8bd9cd1b798e5a197 P fb09133e8ecc409ba9ffc6044c90180f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba716e8ff56743de8c3094b708c833a7, fb09133e8ecc409ba9ffc6044c90180f; no voters:
I20260924 16:12:15.849855 32314 raft_consensus.cc:2781] T dfb42459c86140f8bd9cd1b798e5a197 P fb09133e8ecc409ba9ffc6044c90180f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:12:15.850044 32314 raft_consensus.cc:686] T dfb42459c86140f8bd9cd1b798e5a197 P fb09133e8ecc409ba9ffc6044c90180f [term 1 LEADER]: Becoming Leader. State: Replica: fb09133e8ecc409ba9ffc6044c90180f, State: Running, Role: LEADER
I20260924 16:12:15.850342 32314 consensus_queue.cc:237] T dfb42459c86140f8bd9cd1b798e5a197 P fb09133e8ecc409ba9ffc6044c90180f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f81b244edb0c46c38601587481b42ef0" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 35217 } } peers { permanent_uuid: "fb09133e8ecc409ba9ffc6044c90180f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 44825 } } peers { permanent_uuid: "ba716e8ff56743de8c3094b708c833a7" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 39383 } }
I20260924 16:12:15.853201 31778 catalog_manager.cc:5495] T dfb42459c86140f8bd9cd1b798e5a197 P fb09133e8ecc409ba9ffc6044c90180f reported cstate change: term changed from 0 to 1, leader changed from <none> to fb09133e8ecc409ba9ffc6044c90180f (127.29.234.2). New cstate: current_term: 1 leader_uuid: "fb09133e8ecc409ba9ffc6044c90180f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f81b244edb0c46c38601587481b42ef0" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 35217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb09133e8ecc409ba9ffc6044c90180f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 44825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ba716e8ff56743de8c3094b708c833a7" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 39383 } health_report { overall_health: UNKNOWN } } }
Sep 24 16:12:15 dist-test-slave-v5j9 krb5kdc[31734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266333, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.62@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:12:15.906546 32331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:15.892057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:39697 (local address 127.0.0.1:50734)
0924 16:12:15.892810 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:15.892837 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:15.893011 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:15.893431 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:15.893438 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:15.894145 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:15.894405 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:15.894415 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.895517 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.895522 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:15.896562 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:15.896573 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.897298 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.897304 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:15.897531 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:15.898291 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:15.898314 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:15.900857 (+ 2543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:15.903875 (+ 3018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:15.903890 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:15.903898 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:15.904307 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:15.904667 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:15.904672 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:15.904676 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:15.904811 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:15.905324 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:15.905333 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:15.905528 (+ 195us) client_negotiation.cc:769] Sending connection context
0924 16:12:15.905699 (+ 171us) client_negotiation.cc:240] Negotiation successful
0924 16:12:15.906009 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"mutex_wait_us":1,"thread_start_us":417,"threads_started":1}
I20260924 16:12:15.906569 32332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:15.892078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50734 (local address 127.29.234.62:39697)
0924 16:12:15.893119 (+ 1041us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:15.893127 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:15.893156 (+ 29us) server_negotiation.cc:407] Connection header received
0924 16:12:15.893198 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:15.893201 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:15.893263 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:15.893396 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:15.894572 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.895371 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.896703 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.897119 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.897257 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:15.901050 (+ 3793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:15.901055 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:15.901059 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:15.901085 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:15.903731 (+ 2646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:15.904426 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:15.904433 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:15.904438 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:15.904563 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:15.904918 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:15.904925 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:15.904928 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:15.905196 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:15.905349 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:15.905778 (+ 429us) server_negotiation.cc:299] Negotiation successful
0924 16:12:15.906121 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":160,"server-negotiator.queue_time_us":912,"thread_start_us":772,"threads_started":1}
I20260924 16:12:15.912604 31778 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.29.234.62@KRBTEST.COM'} at 127.0.0.1:50734:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 16:12:15.913841 31778 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:12:15.920734 31918 tablet_service.cc:1467] Processing CreateTablet for tablet 72f22fadca64477c990a973201ce0166 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e7ec650004134056a1b0ed57920a1da7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 16:12:15.920867 32058 tablet_service.cc:1467] Processing CreateTablet for tablet 72f22fadca64477c990a973201ce0166 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e7ec650004134056a1b0ed57920a1da7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 16:12:15.921062 32205 tablet_service.cc:1467] Processing CreateTablet for tablet 72f22fadca64477c990a973201ce0166 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e7ec650004134056a1b0ed57920a1da7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 16:12:15.921311 31918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72f22fadca64477c990a973201ce0166. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:15.921448 32058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72f22fadca64477c990a973201ce0166. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:15.921598 32205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72f22fadca64477c990a973201ce0166. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:15.928694 32297 tablet_bootstrap.cc:492] T 72f22fadca64477c990a973201ce0166 P ba716e8ff56743de8c3094b708c833a7: Bootstrap starting.
I20260924 16:12:15.928705 32296 tablet_bootstrap.cc:492] T 72f22fadca64477c990a973201ce0166 P f81b244edb0c46c38601587481b42ef0: Bootstrap starting.
I20260924 16:12:15.930423 32296 tablet_bootstrap.cc:654] T 72f22fadca64477c990a973201ce0166 P f81b244edb0c46c38601587481b42ef0: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:15.930423 32297 tablet_bootstrap.cc:654] T 72f22fadca64477c990a973201ce0166 P ba716e8ff56743de8c3094b708c833a7: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:15.931335 32295 tablet_bootstrap.cc:492] T 72f22fadca64477c990a973201ce0166 P fb09133e8ecc409ba9ffc6044c90180f: Bootstrap starting.
I20260924 16:12:15.934707 32295 tablet_bootstrap.cc:654] T 72f22fadca64477c990a973201ce0166 P fb09133e8ecc409ba9ffc6044c90180f: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:15.938396 32296 tablet_bootstrap.cc:492] T 72f22fadca64477c990a973201ce0166 P f81b244edb0c46c38601587481b42ef0: No bootstrap required, opened a new log
I20260924 16:12:15.938573 32296 ts_tablet_manager.cc:1397] T 72f22fadca64477c990a973201ce0166 P f81b244edb0c46c38601587481b42ef0: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.005s
I20260924 16:12:15.939113 32295 tablet_bootstrap.cc:492] T 72f22fadca64477c990a973201ce0166 P fb09133e8ecc409ba9ffc6044c90180f: No bootstrap required, opened a new log
I20260924 16:12:15.939308 32295 ts_tablet_manager.cc:1397] T 72f22fadca64477c990a973201ce0166 P fb09133e8ecc409ba9ffc6044c90180f: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.002s
I20260924 16:12:15.939388 32296 raft_consensus.cc:348] T 72f22fadca64477c990a973201ce0166 P f81b244edb0c46c38601587481b42ef0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f81b244edb0c46c38601587481b42ef0" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 35217 } } peers { permanent_uuid: "fb09133e8ecc409ba9ffc6044c90180f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 44825 } } peers { permanent_uuid: "ba716e8ff56743de8c3094b708c833a7" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 39383 } }
I20260924 16:12:15.939651 32296 raft_consensus.cc:374] T 72f22fadca64477c990a973201ce0166 P f81b244edb0c46c38601587481b42ef0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:15.939744 32296 raft_consensus.cc:729] T 72f22fadca64477c990a973201ce0166 P f81b244edb0c46c38601587481b42ef0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f81b244edb0c46c38601587481b42ef0, State: Initialized, Role: FOLLOWER
I20260924 16:12:15.939988 32296 consensus_queue.cc:260] T 72f22fadca64477c990a973201ce0166 P f81b244edb0c46c38601587481b42ef0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f81b244edb0c46c38601587481b42ef0" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 35217 } } peers { permanent_uuid: "fb09133e8ecc409ba9ffc6044c90180f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 44825 } } peers { permanent_uuid: "ba716e8ff56743de8c3094b708c833a7" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 39383 } }
I20260924 16:12:15.940158 32295 raft_consensus.cc:348] T 72f22fadca64477c990a973201ce0166 P fb09133e8ecc409ba9ffc6044c90180f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f81b244edb0c46c38601587481b42ef0" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 35217 } } peers { permanent_uuid: "fb09133e8ecc409ba9ffc6044c90180f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 44825 } } peers { permanent_uuid: "ba716e8ff56743de8c3094b708c833a7" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 39383 } }
I20260924 16:12:15.940407 32295 raft_consensus.cc:374] T 72f22fadca64477c990a973201ce0166 P fb09133e8ecc409ba9ffc6044c90180f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:15.940506 32295 raft_consensus.cc:729] T 72f22fadca64477c990a973201ce0166 P fb09133e8ecc409ba9ffc6044c90180f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb09133e8ecc409ba9ffc6044c90180f, State: Initialized, Role: FOLLOWER
I20260924 16:12:15.940450 32302 tablet_replica.cc:439] T 72f22fadca64477c990a973201ce0166 P f81b244edb0c46c38601587481b42ef0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f81b244edb0c46c38601587481b42ef0" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 35217 } } peers { permanent_uuid: "fb09133e8ecc409ba9ffc6044c90180f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 44825 } } peers { permanent_uuid: "ba716e8ff56743de8c3094b708c833a7" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 39383 } } }
I20260924 16:12:15.940788 32302 tablet_replica.cc:442] T 72f22fadca64477c990a973201ce0166 P f81b244edb0c46c38601587481b42ef0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:12:15.940779 32295 consensus_queue.cc:260] T 72f22fadca64477c990a973201ce0166 P fb09133e8ecc409ba9ffc6044c90180f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f81b244edb0c46c38601587481b42ef0" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 35217 } } peers { permanent_uuid: "fb09133e8ecc409ba9ffc6044c90180f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 44825 } } peers { permanent_uuid: "ba716e8ff56743de8c3094b708c833a7" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 39383 } }
I20260924 16:12:15.941246 32314 tablet_replica.cc:439] T 72f22fadca64477c990a973201ce0166 P fb09133e8ecc409ba9ffc6044c90180f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f81b244edb0c46c38601587481b42ef0" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 35217 } } peers { permanent_uuid: "fb09133e8ecc409ba9ffc6044c90180f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 44825 } } peers { permanent_uuid: "ba716e8ff56743de8c3094b708c833a7" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 39383 } } }
I20260924 16:12:15.941432 32295 ts_tablet_manager.cc:1428] T 72f22fadca64477c990a973201ce0166 P fb09133e8ecc409ba9ffc6044c90180f: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260924 16:12:15.941566 32314 tablet_replica.cc:442] T 72f22fadca64477c990a973201ce0166 P fb09133e8ecc409ba9ffc6044c90180f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:12:15.942056 32296 ts_tablet_manager.cc:1428] T 72f22fadca64477c990a973201ce0166 P f81b244edb0c46c38601587481b42ef0: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260924 16:12:15.943512 32297 tablet_bootstrap.cc:492] T 72f22fadca64477c990a973201ce0166 P ba716e8ff56743de8c3094b708c833a7: No bootstrap required, opened a new log
I20260924 16:12:15.943724 32297 ts_tablet_manager.cc:1397] T 72f22fadca64477c990a973201ce0166 P ba716e8ff56743de8c3094b708c833a7: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.000s
I20260924 16:12:15.944532 32297 raft_consensus.cc:348] T 72f22fadca64477c990a973201ce0166 P ba716e8ff56743de8c3094b708c833a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f81b244edb0c46c38601587481b42ef0" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 35217 } } peers { permanent_uuid: "fb09133e8ecc409ba9ffc6044c90180f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 44825 } } peers { permanent_uuid: "ba716e8ff56743de8c3094b708c833a7" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 39383 } }
I20260924 16:12:15.944753 32297 raft_consensus.cc:374] T 72f22fadca64477c990a973201ce0166 P ba716e8ff56743de8c3094b708c833a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:15.944856 32297 raft_consensus.cc:729] T 72f22fadca64477c990a973201ce0166 P ba716e8ff56743de8c3094b708c833a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba716e8ff56743de8c3094b708c833a7, State: Initialized, Role: FOLLOWER
I20260924 16:12:15.945103 32297 consensus_queue.cc:260] T 72f22fadca64477c990a973201ce0166 P ba716e8ff56743de8c3094b708c833a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f81b244edb0c46c38601587481b42ef0" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 35217 } } peers { permanent_uuid: "fb09133e8ecc409ba9ffc6044c90180f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 44825 } } peers { permanent_uuid: "ba716e8ff56743de8c3094b708c833a7" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 39383 } }
I20260924 16:12:15.945696 32297 ts_tablet_manager.cc:1428] T 72f22fadca64477c990a973201ce0166 P ba716e8ff56743de8c3094b708c833a7: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 16:12:15.945591 32303 tablet_replica.cc:439] T 72f22fadca64477c990a973201ce0166 P ba716e8ff56743de8c3094b708c833a7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f81b244edb0c46c38601587481b42ef0" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 35217 } } peers { permanent_uuid: "fb09133e8ecc409ba9ffc6044c90180f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 44825 } } peers { permanent_uuid: "ba716e8ff56743de8c3094b708c833a7" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 39383 } } }
I20260924 16:12:15.945935 32303 tablet_replica.cc:442] T 72f22fadca64477c990a973201ce0166 P ba716e8ff56743de8c3094b708c833a7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:12:15.967612 32303 raft_consensus.cc:482] T 72f22fadca64477c990a973201ce0166 P ba716e8ff56743de8c3094b708c833a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:12:15.967861 32303 raft_consensus.cc:504] T 72f22fadca64477c990a973201ce0166 P ba716e8ff56743de8c3094b708c833a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f81b244edb0c46c38601587481b42ef0" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 35217 } } peers { permanent_uuid: "fb09133e8ecc409ba9ffc6044c90180f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 44825 } } peers { permanent_uuid: "ba716e8ff56743de8c3094b708c833a7" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 39383 } }
I20260924 16:12:15.968662 32302 raft_consensus.cc:482] T 72f22fadca64477c990a973201ce0166 P f81b244edb0c46c38601587481b42ef0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:12:15.968930 32302 raft_consensus.cc:504] T 72f22fadca64477c990a973201ce0166 P f81b244edb0c46c38601587481b42ef0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f81b244edb0c46c38601587481b42ef0" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 35217 } } peers { permanent_uuid: "fb09133e8ecc409ba9ffc6044c90180f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 44825 } } peers { permanent_uuid: "ba716e8ff56743de8c3094b708c833a7" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 39383 } }
I20260924 16:12:15.970175 32303 leader_election.cc:288] T 72f22fadca64477c990a973201ce0166 P ba716e8ff56743de8c3094b708c833a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f81b244edb0c46c38601587481b42ef0 (127.29.234.1:35217), fb09133e8ecc409ba9ffc6044c90180f (127.29.234.2:44825)
I20260924 16:12:15.970830 32302 leader_election.cc:288] T 72f22fadca64477c990a973201ce0166 P f81b244edb0c46c38601587481b42ef0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb09133e8ecc409ba9ffc6044c90180f (127.29.234.2:44825), ba716e8ff56743de8c3094b708c833a7 (127.29.234.3:39383)
I20260924 16:12:15.985095 32288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:15.970662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.3:48789 (local address 127.29.234.1:35217)
0924 16:12:15.970870 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:15.970877 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:15.973876 (+ 2999us) server_negotiation.cc:407] Connection header received
0924 16:12:15.974187 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:15.974196 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:15.974285 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:15.974355 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:12:15.975399 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.976968 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.980264 (+ 3296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.982116 (+ 1852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.982317 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:15.982436 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:15.983907 (+ 1471us) server_negotiation.cc:299] Negotiation successful
0924 16:12:15.984050 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 16:12:15.985518 32336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:15.970237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:35217 (local address 127.29.234.3:48789)
0924 16:12:15.973759 (+ 3522us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:15.973783 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:15.974017 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:15.974670 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:15.974677 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:15.974714 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:12:15.975148 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:15.975166 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.977291 (+ 2125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.977299 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:15.980083 (+ 2784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:15.980098 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.983380 (+ 3282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.983390 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:15.983676 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:15.983728 (+ 52us) client_negotiation.cc:769] Sending connection context
0924 16:12:15.984895 (+ 1167us) client_negotiation.cc:240] Negotiation successful
0924 16:12:15.985119 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3308,"thread_start_us":3120,"threads_started":1}
I20260924 16:12:15.986289 31944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72f22fadca64477c990a973201ce0166" candidate_uuid: "ba716e8ff56743de8c3094b708c833a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f81b244edb0c46c38601587481b42ef0" is_pre_election: true
I20260924 16:12:15.986690 31944 raft_consensus.cc:2454] T 72f22fadca64477c990a973201ce0166 P f81b244edb0c46c38601587481b42ef0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba716e8ff56743de8c3094b708c833a7 in term 0.
I20260924 16:12:15.987385 32154 leader_election.cc:302] T 72f22fadca64477c990a973201ce0166 P ba716e8ff56743de8c3094b708c833a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba716e8ff56743de8c3094b708c833a7, f81b244edb0c46c38601587481b42ef0; no voters:
I20260924 16:12:15.987874 32303 raft_consensus.cc:2781] T 72f22fadca64477c990a973201ce0166 P ba716e8ff56743de8c3094b708c833a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:12:15.987988 32303 raft_consensus.cc:482] T 72f22fadca64477c990a973201ce0166 P ba716e8ff56743de8c3094b708c833a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:12:15.988073 32303 raft_consensus.cc:3037] T 72f22fadca64477c990a973201ce0166 P ba716e8ff56743de8c3094b708c833a7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:15.991526 32303 raft_consensus.cc:504] T 72f22fadca64477c990a973201ce0166 P ba716e8ff56743de8c3094b708c833a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f81b244edb0c46c38601587481b42ef0" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 35217 } } peers { permanent_uuid: "fb09133e8ecc409ba9ffc6044c90180f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 44825 } } peers { permanent_uuid: "ba716e8ff56743de8c3094b708c833a7" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 39383 } }
I20260924 16:12:15.992893 31944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72f22fadca64477c990a973201ce0166" candidate_uuid: "ba716e8ff56743de8c3094b708c833a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f81b244edb0c46c38601587481b42ef0"
I20260924 16:12:15.993139 31944 raft_consensus.cc:3037] T 72f22fadca64477c990a973201ce0166 P f81b244edb0c46c38601587481b42ef0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:15.994825 32294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:15.970843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.3:44073 (local address 127.29.234.2:44825)
0924 16:12:15.971064 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:15.971070 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:15.984537 (+ 13467us) server_negotiation.cc:407] Connection header received
0924 16:12:15.984606 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:15.984611 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:15.984738 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:15.984833 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:12:15.987999 (+ 3166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.989126 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.992072 (+ 2946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.993532 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.993663 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:15.993741 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:15.994348 (+ 607us) server_negotiation.cc:299] Negotiation successful
0924 16:12:15.994503 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 16:12:15.994993 32291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:15.973664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.1:32813 (local address 127.29.234.3:39383)
0924 16:12:15.973833 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:15.973840 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:15.983353 (+ 9513us) server_negotiation.cc:407] Connection header received
0924 16:12:15.983533 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:15.983540 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:15.983747 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:15.983861 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:12:15.987270 (+ 3409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.988493 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.990884 (+ 2391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.992434 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.992554 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:15.992628 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:15.994599 (+ 1971us) server_negotiation.cc:299] Negotiation successful
0924 16:12:15.994734 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":58}
I20260924 16:12:15.995599 32338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:15.971389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.2:44825 (local address 127.29.234.3:44073)
0924 16:12:15.984089 (+ 12700us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:15.984112 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:15.984320 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:15.984868 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:15.984875 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:15.984903 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:12:15.985283 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:15.985295 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.989328 (+ 4033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.989335 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:15.991876 (+ 2541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:15.991891 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.993939 (+ 2048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.993945 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:15.994138 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:15.994171 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 16:12:15.995073 (+ 902us) client_negotiation.cc:240] Negotiation successful
0924 16:12:15.995230 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12576,"mutex_wait_us":42,"thread_start_us":13086,"threads_started":1}
I20260924 16:12:15.996097 32339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:15.971626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.3:39383 (local address 127.29.234.1:32813)
0924 16:12:15.983241 (+ 11615us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:15.983267 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:15.983416 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:15.984071 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:15.984077 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:15.984106 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:12:15.987077 (+ 2971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:15.987091 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.988757 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.988763 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:15.990713 (+ 1950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:15.990729 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.994176 (+ 3447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.994183 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:15.994396 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:15.994434 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 16:12:15.995466 (+ 1032us) client_negotiation.cc:240] Negotiation successful
0924 16:12:15.995610 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11419,"thread_start_us":11098,"threads_started":1}
I20260924 16:12:15.996227 31944 raft_consensus.cc:2454] T 72f22fadca64477c990a973201ce0166 P f81b244edb0c46c38601587481b42ef0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba716e8ff56743de8c3094b708c833a7 in term 1.
I20260924 16:12:15.996528 32081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72f22fadca64477c990a973201ce0166" candidate_uuid: "ba716e8ff56743de8c3094b708c833a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb09133e8ecc409ba9ffc6044c90180f" is_pre_election: true
I20260924 16:12:15.996843 32154 leader_election.cc:302] T 72f22fadca64477c990a973201ce0166 P ba716e8ff56743de8c3094b708c833a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba716e8ff56743de8c3094b708c833a7, f81b244edb0c46c38601587481b42ef0; no voters:
I20260924 16:12:15.997057 32081 raft_consensus.cc:2454] T 72f22fadca64477c990a973201ce0166 P fb09133e8ecc409ba9ffc6044c90180f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba716e8ff56743de8c3094b708c833a7 in term 0.
I20260924 16:12:15.997321 32226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72f22fadca64477c990a973201ce0166" candidate_uuid: "f81b244edb0c46c38601587481b42ef0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba716e8ff56743de8c3094b708c833a7" is_pre_election: true
I20260924 16:12:15.997673 32226 raft_consensus.cc:2379] T 72f22fadca64477c990a973201ce0166 P ba716e8ff56743de8c3094b708c833a7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f81b244edb0c46c38601587481b42ef0 in current term 1: Already voted for candidate ba716e8ff56743de8c3094b708c833a7 in this term.
I20260924 16:12:15.999663 32303 leader_election.cc:288] T 72f22fadca64477c990a973201ce0166 P ba716e8ff56743de8c3094b708c833a7 [CANDIDATE]: Term 1 election: Requested vote from peers f81b244edb0c46c38601587481b42ef0 (127.29.234.1:35217), fb09133e8ecc409ba9ffc6044c90180f (127.29.234.2:44825)
I20260924 16:12:16.000022 32303 raft_consensus.cc:2781] T 72f22fadca64477c990a973201ce0166 P ba716e8ff56743de8c3094b708c833a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:12:16.000072 32337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:15.971414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.2:44825 (local address 127.29.234.1:47325)
0924 16:12:15.972845 (+ 1431us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:15.972872 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:15.973049 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:15.982515 (+ 9466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:15.982522 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:15.982578 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:12:15.983151 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:15.983165 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.985483 (+ 2318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.985488 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:15.987800 (+ 2312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:15.987814 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.999222 (+ 11408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.999229 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:15.999426 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:15.999463 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 16:12:15.999564 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 16:12:15.999717 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1255,"thread_start_us":1061,"threads_started":1}
I20260924 16:12:16.000411 32303 raft_consensus.cc:686] T 72f22fadca64477c990a973201ce0166 P ba716e8ff56743de8c3094b708c833a7 [term 1 LEADER]: Becoming Leader. State: Replica: ba716e8ff56743de8c3094b708c833a7, State: Running, Role: LEADER
I20260924 16:12:16.000802 32081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72f22fadca64477c990a973201ce0166" candidate_uuid: "ba716e8ff56743de8c3094b708c833a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb09133e8ecc409ba9ffc6044c90180f"
I20260924 16:12:16.001050 32081 raft_consensus.cc:3037] T 72f22fadca64477c990a973201ce0166 P fb09133e8ecc409ba9ffc6044c90180f [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:16.001168 32303 consensus_queue.cc:237] T 72f22fadca64477c990a973201ce0166 P ba716e8ff56743de8c3094b708c833a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f81b244edb0c46c38601587481b42ef0" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 35217 } } peers { permanent_uuid: "fb09133e8ecc409ba9ffc6044c90180f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 44825 } } peers { permanent_uuid: "ba716e8ff56743de8c3094b708c833a7" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 39383 } }
I20260924 16:12:16.004524 32081 raft_consensus.cc:2454] T 72f22fadca64477c990a973201ce0166 P fb09133e8ecc409ba9ffc6044c90180f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba716e8ff56743de8c3094b708c833a7 in term 1.
I20260924 16:12:16.004591 32341 tablet_replica.cc:439] T 72f22fadca64477c990a973201ce0166 P ba716e8ff56743de8c3094b708c833a7: TxnStatusTablet state changed. Reason: New leader ba716e8ff56743de8c3094b708c833a7. Latest consensus state: current_term: 1 leader_uuid: "ba716e8ff56743de8c3094b708c833a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f81b244edb0c46c38601587481b42ef0" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 35217 } } peers { permanent_uuid: "fb09133e8ecc409ba9ffc6044c90180f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 44825 } } peers { permanent_uuid: "ba716e8ff56743de8c3094b708c833a7" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 39383 } } }
I20260924 16:12:16.005151 32341 tablet_replica.cc:442] T 72f22fadca64477c990a973201ce0166 P ba716e8ff56743de8c3094b708c833a7: This TxnStatusTablet replica's current role is: LEADER
I20260924 16:12:16.005821 32340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:15.971988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.1:47325 (local address 127.29.234.2:44825)
0924 16:12:15.981881 (+ 9893us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:15.981890 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:15.981923 (+ 33us) server_negotiation.cc:407] Connection header received
0924 16:12:15.982016 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:15.982024 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:15.982129 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:15.982265 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:12:15.983494 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.984741 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.996665 (+ 11924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:15.998971 (+ 2306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:15.999241 (+ 270us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:15.999369 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:16.005250 (+ 5881us) server_negotiation.cc:299] Negotiation successful
0924 16:12:16.005444 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9715,"thread_start_us":9525,"threads_started":1}
I20260924 16:12:16.006292 32081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72f22fadca64477c990a973201ce0166" candidate_uuid: "f81b244edb0c46c38601587481b42ef0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb09133e8ecc409ba9ffc6044c90180f" is_pre_election: true
I20260924 16:12:16.006703 32081 raft_consensus.cc:2379] T 72f22fadca64477c990a973201ce0166 P fb09133e8ecc409ba9ffc6044c90180f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f81b244edb0c46c38601587481b42ef0 in current term 1: Already voted for candidate ba716e8ff56743de8c3094b708c833a7 in this term.
I20260924 16:12:16.007493 31869 leader_election.cc:302] T 72f22fadca64477c990a973201ce0166 P f81b244edb0c46c38601587481b42ef0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f81b244edb0c46c38601587481b42ef0; no voters: ba716e8ff56743de8c3094b708c833a7, fb09133e8ecc409ba9ffc6044c90180f
I20260924 16:12:16.007166 31778 catalog_manager.cc:5495] T 72f22fadca64477c990a973201ce0166 P ba716e8ff56743de8c3094b708c833a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to ba716e8ff56743de8c3094b708c833a7 (127.29.234.3). New cstate: current_term: 1 leader_uuid: "ba716e8ff56743de8c3094b708c833a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f81b244edb0c46c38601587481b42ef0" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 35217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb09133e8ecc409ba9ffc6044c90180f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 44825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ba716e8ff56743de8c3094b708c833a7" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 39383 } health_report { overall_health: HEALTHY } } }
I20260924 16:12:16.008497 32302 raft_consensus.cc:2726] T 72f22fadca64477c990a973201ce0166 P f81b244edb0c46c38601587481b42ef0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 16:12:16.012899 32343 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 16:12:16.013146 32343 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 16:12:16.044757 32331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:16.038616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.3:39383 (local address 127.0.0.1:42638)
0924 16:12:16.038966 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:16.038982 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:16.039148 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:16.039643 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:16.039648 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:16.039660 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 16:12:16.039938 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:16.039949 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:16.040974 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:16.040979 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:16.042283 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:16.042296 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:16.043038 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:16.043044 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:16.043203 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:16.043259 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 16:12:16.044195 (+ 936us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 16:12:16.044218 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 16:12:16.044303 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 16:12:16.044457 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229}
I20260924 16:12:16.044755 32291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:16.038666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42638 (local address 127.29.234.3:39383)
0924 16:12:16.038862 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:16.038870 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:16.039061 (+ 191us) server_negotiation.cc:407] Connection header received
0924 16:12:16.039362 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:16.039366 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:16.039473 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:16.039620 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:12:16.040079 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:16.040842 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:16.042432 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:16.042875 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:16.043024 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:16.043458 (+ 434us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:12:16.044083 (+ 625us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:12:16.044236 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:16.044321 (+ 85us) server_negotiation.cc:299] Negotiation successful
0924 16:12:16.044428 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 16:12:16.049271 31944 raft_consensus.cc:1261] T 72f22fadca64477c990a973201ce0166 P f81b244edb0c46c38601587481b42ef0 [term 1 FOLLOWER]: Refusing update from remote peer ba716e8ff56743de8c3094b708c833a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:12:16.049400 32081 raft_consensus.cc:1261] T 72f22fadca64477c990a973201ce0166 P fb09133e8ecc409ba9ffc6044c90180f [term 1 FOLLOWER]: Refusing update from remote peer ba716e8ff56743de8c3094b708c833a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:12:16.050141 32303 consensus_queue.cc:1035] T 72f22fadca64477c990a973201ce0166 P ba716e8ff56743de8c3094b708c833a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb09133e8ecc409ba9ffc6044c90180f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 44825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:12:16.050525 32341 consensus_queue.cc:1035] T 72f22fadca64477c990a973201ce0166 P ba716e8ff56743de8c3094b708c833a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f81b244edb0c46c38601587481b42ef0" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 35217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:12:16.054339 32302 tablet_replica.cc:439] T 72f22fadca64477c990a973201ce0166 P f81b244edb0c46c38601587481b42ef0: TxnStatusTablet state changed. Reason: New leader ba716e8ff56743de8c3094b708c833a7. Latest consensus state: current_term: 1 leader_uuid: "ba716e8ff56743de8c3094b708c833a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f81b244edb0c46c38601587481b42ef0" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 35217 } } peers { permanent_uuid: "fb09133e8ecc409ba9ffc6044c90180f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 44825 } } peers { permanent_uuid: "ba716e8ff56743de8c3094b708c833a7" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 39383 } } }
I20260924 16:12:16.054741 32302 tablet_replica.cc:442] T 72f22fadca64477c990a973201ce0166 P f81b244edb0c46c38601587481b42ef0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:12:16.055521 32314 tablet_replica.cc:439] T 72f22fadca64477c990a973201ce0166 P fb09133e8ecc409ba9ffc6044c90180f: TxnStatusTablet state changed. Reason: New leader ba716e8ff56743de8c3094b708c833a7. Latest consensus state: current_term: 1 leader_uuid: "ba716e8ff56743de8c3094b708c833a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f81b244edb0c46c38601587481b42ef0" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 35217 } } peers { permanent_uuid: "fb09133e8ecc409ba9ffc6044c90180f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 44825 } } peers { permanent_uuid: "ba716e8ff56743de8c3094b708c833a7" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 39383 } } }
I20260924 16:12:16.055914 32314 tablet_replica.cc:442] T 72f22fadca64477c990a973201ce0166 P fb09133e8ecc409ba9ffc6044c90180f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:12:16.059224 32341 tablet_replica.cc:439] T 72f22fadca64477c990a973201ce0166 P ba716e8ff56743de8c3094b708c833a7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ba716e8ff56743de8c3094b708c833a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f81b244edb0c46c38601587481b42ef0" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 35217 } } peers { permanent_uuid: "fb09133e8ecc409ba9ffc6044c90180f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 44825 } } peers { permanent_uuid: "ba716e8ff56743de8c3094b708c833a7" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 39383 } } }
I20260924 16:12:16.059526 32341 tablet_replica.cc:442] T 72f22fadca64477c990a973201ce0166 P ba716e8ff56743de8c3094b708c833a7: This TxnStatusTablet replica's current role is: LEADER
I20260924 16:12:16.059897 32310 mvcc.cc:205] Tried to move back new op lower bound from 7332930912449945600 to 7332930912267370496. Current Snapshot: MvccSnapshot[applied={T|T < 7332930912449945600 or (T in {7332930912449945600})}]
I20260924 16:12:16.062693 32314 tablet_replica.cc:439] T 72f22fadca64477c990a973201ce0166 P fb09133e8ecc409ba9ffc6044c90180f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ba716e8ff56743de8c3094b708c833a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f81b244edb0c46c38601587481b42ef0" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 35217 } } peers { permanent_uuid: "fb09133e8ecc409ba9ffc6044c90180f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 44825 } } peers { permanent_uuid: "ba716e8ff56743de8c3094b708c833a7" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 39383 } } }
I20260924 16:12:16.063026 32314 tablet_replica.cc:442] T 72f22fadca64477c990a973201ce0166 P fb09133e8ecc409ba9ffc6044c90180f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:12:16.085173 32302 tablet_replica.cc:439] T 72f22fadca64477c990a973201ce0166 P f81b244edb0c46c38601587481b42ef0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ba716e8ff56743de8c3094b708c833a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f81b244edb0c46c38601587481b42ef0" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 35217 } } peers { permanent_uuid: "fb09133e8ecc409ba9ffc6044c90180f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 44825 } } peers { permanent_uuid: "ba716e8ff56743de8c3094b708c833a7" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 39383 } } }
I20260924 16:12:16.085879 32302 tablet_replica.cc:442] T 72f22fadca64477c990a973201ce0166 P f81b244edb0c46c38601587481b42ef0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:12:16.085666 32311 mvcc.cc:205] Tried to move back new op lower bound from 7332930912449945600 to 7332930912267370496. Current Snapshot: MvccSnapshot[applied={T|T < 7332930912449945600}]
I20260924 16:12:16.091750 32348 tablet_replica.cc:439] T 72f22fadca64477c990a973201ce0166 P ba716e8ff56743de8c3094b708c833a7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ba716e8ff56743de8c3094b708c833a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f81b244edb0c46c38601587481b42ef0" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 35217 } } peers { permanent_uuid: "fb09133e8ecc409ba9ffc6044c90180f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 44825 } } peers { permanent_uuid: "ba716e8ff56743de8c3094b708c833a7" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 39383 } } }
I20260924 16:12:16.092168 32348 tablet_replica.cc:442] T 72f22fadca64477c990a973201ce0166 P ba716e8ff56743de8c3094b708c833a7: This TxnStatusTablet replica's current role is: LEADER
I20260924 16:12:16.109155 32291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:16.101680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.2:55189 (local address 127.29.234.3:39383)
0924 16:12:16.101904 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:16.101911 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:16.102167 (+ 256us) server_negotiation.cc:407] Connection header received
0924 16:12:16.102466 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:16.102474 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:16.102569 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:16.102690 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:12:16.103240 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:16.104484 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:16.106798 (+ 2314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:16.107567 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:16.107737 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:16.108271 (+ 534us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:12:16.108442 (+ 171us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:12:16.108551 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:16.108731 (+ 180us) server_negotiation.cc:299] Negotiation successful
0924 16:12:16.108819 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 16:12:16.109352 32352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:16.100934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.3:39383 (local address 127.29.234.2:55189)
0924 16:12:16.102055 (+ 1121us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:16.102075 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:16.102265 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:16.102714 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:16.102719 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:16.102736 (+ 17us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 16:12:16.103054 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:16.103068 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:16.104656 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:16.104662 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:16.106630 (+ 1968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:16.106646 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:16.107774 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:16.107781 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:16.107978 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:16.108062 (+ 84us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 16:12:16.108566 (+ 504us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 16:12:16.108578 (+ 12us) client_negotiation.cc:769] Sending connection context
0924 16:12:16.108669 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 16:12:16.108820 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":937,"thread_start_us":839,"threads_started":1}
I20260924 16:12:16.122077 31944 raft_consensus.cc:1261] T dfb42459c86140f8bd9cd1b798e5a197 P f81b244edb0c46c38601587481b42ef0 [term 1 FOLLOWER]: Refusing update from remote peer fb09133e8ecc409ba9ffc6044c90180f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:12:16.122107 32226 raft_consensus.cc:1261] T dfb42459c86140f8bd9cd1b798e5a197 P ba716e8ff56743de8c3094b708c833a7 [term 1 FOLLOWER]: Refusing update from remote peer fb09133e8ecc409ba9ffc6044c90180f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:12:16.122740 32301 consensus_queue.cc:1035] T dfb42459c86140f8bd9cd1b798e5a197 P fb09133e8ecc409ba9ffc6044c90180f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f81b244edb0c46c38601587481b42ef0" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 35217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:12:16.123225 32301 consensus_queue.cc:1035] T dfb42459c86140f8bd9cd1b798e5a197 P fb09133e8ecc409ba9ffc6044c90180f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba716e8ff56743de8c3094b708c833a7" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 39383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:12:16.141440 32309 mvcc.cc:205] Tried to move back new op lower bound from 7332930912749875200 to 7332930911645622272. Current Snapshot: MvccSnapshot[applied={T|T < 7332930912749875200}]
I20260924 16:12:16.239106 32340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:16.211301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.3:49315 (local address 127.29.234.2:44825)
0924 16:12:16.211517 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:16.211525 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:16.229065 (+ 17540us) server_negotiation.cc:407] Connection header received
0924 16:12:16.229830 (+ 765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:16.229838 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:16.229934 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:16.229999 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:12:16.230905 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:16.232119 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:16.236741 (+ 4622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:16.237431 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:16.237531 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:16.238248 (+ 717us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:12:16.238402 (+ 154us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:12:16.238466 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:16.238699 (+ 233us) server_negotiation.cc:299] Negotiation successful
0924 16:12:16.238793 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 16:12:16.239992 32356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:16.210833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.2:44825 (local address 127.29.234.3:49315)
0924 16:12:16.228954 (+ 18121us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:16.228975 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:16.229616 (+ 641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:16.230229 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:16.230234 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:16.230246 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 16:12:16.230718 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:16.230730 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:16.232309 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:16.232314 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:16.234269 (+ 1955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:16.234285 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:16.237745 (+ 3460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:16.237752 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:16.237931 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:16.238018 (+ 87us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 16:12:16.238552 (+ 534us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 16:12:16.238563 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 16:12:16.239383 (+ 820us) client_negotiation.cc:240] Negotiation successful
0924 16:12:16.239545 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17871,"thread_start_us":18255,"threads_started":1}
I20260924 16:12:16.282091 31778 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50730:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:12:16.282377 31778 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50730:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:12:16.286077 31778 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dde4f26be06a4b1799e168313c156528: Sending DeleteTablet for 3 replicas of tablet dfb42459c86140f8bd9cd1b798e5a197
I20260924 16:12:16.286922 32058 tablet_service.cc:1514] Processing DeleteTablet for tablet dfb42459c86140f8bd9cd1b798e5a197 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:12:16 UTC) from {username='kudu', principal='kudu/127.29.234.62@KRBTEST.COM'} at 127.0.0.1:50374
I20260924 16:12:16.287243 31918 tablet_service.cc:1514] Processing DeleteTablet for tablet dfb42459c86140f8bd9cd1b798e5a197 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:12:16 UTC) from {username='kudu', principal='kudu/127.29.234.62@KRBTEST.COM'} at 127.0.0.1:58496
I20260924 16:12:16.287496 32205 tablet_service.cc:1514] Processing DeleteTablet for tablet dfb42459c86140f8bd9cd1b798e5a197 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:12:16 UTC) from {username='kudu', principal='kudu/127.29.234.62@KRBTEST.COM'} at 127.0.0.1:42636
I20260924 16:12:16.302603 32359 tablet_replica.cc:330] T dfb42459c86140f8bd9cd1b798e5a197 P f81b244edb0c46c38601587481b42ef0: stopping tablet replica
I20260924 16:12:16.302853 32359 raft_consensus.cc:2229] T dfb42459c86140f8bd9cd1b798e5a197 P f81b244edb0c46c38601587481b42ef0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:12:16.303130 32359 raft_consensus.cc:2258] T dfb42459c86140f8bd9cd1b798e5a197 P f81b244edb0c46c38601587481b42ef0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:12:16.304505 32359 ts_tablet_manager.cc:1905] T dfb42459c86140f8bd9cd1b798e5a197 P f81b244edb0c46c38601587481b42ef0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:12:16.309623 32360 tablet_replica.cc:330] T dfb42459c86140f8bd9cd1b798e5a197 P ba716e8ff56743de8c3094b708c833a7: stopping tablet replica
I20260924 16:12:16.310007 32360 raft_consensus.cc:2229] T dfb42459c86140f8bd9cd1b798e5a197 P ba716e8ff56743de8c3094b708c833a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:12:16.310380 32360 raft_consensus.cc:2258] T dfb42459c86140f8bd9cd1b798e5a197 P ba716e8ff56743de8c3094b708c833a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:12:16.312194 32360 ts_tablet_manager.cc:1905] T dfb42459c86140f8bd9cd1b798e5a197 P ba716e8ff56743de8c3094b708c833a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:12:16.314569 32359 ts_tablet_manager.cc:1918] T dfb42459c86140f8bd9cd1b798e5a197 P f81b244edb0c46c38601587481b42ef0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 16:12:16.314724 32359 log.cc:1196] T dfb42459c86140f8bd9cd1b798e5a197 P f81b244edb0c46c38601587481b42ef0: Deleting WAL directory at /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/minicluster-data/ts-0/wal/wals/dfb42459c86140f8bd9cd1b798e5a197
I20260924 16:12:16.315176 32359 ts_tablet_manager.cc:1939] T dfb42459c86140f8bd9cd1b798e5a197 P f81b244edb0c46c38601587481b42ef0: Deleting consensus metadata
I20260924 16:12:16.315735 32358 tablet_replica.cc:330] T dfb42459c86140f8bd9cd1b798e5a197 P fb09133e8ecc409ba9ffc6044c90180f: stopping tablet replica
I20260924 16:12:16.316377 31765 catalog_manager.cc:4841] TS f81b244edb0c46c38601587481b42ef0 (127.29.234.1:35217): tablet dfb42459c86140f8bd9cd1b798e5a197 (table test-table [id=37c4b5c5ec534d22997f524fd6ca8d93]) successfully deleted
I20260924 16:12:16.316746 32358 raft_consensus.cc:2229] T dfb42459c86140f8bd9cd1b798e5a197 P fb09133e8ecc409ba9ffc6044c90180f [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:12:16.317140 32358 raft_consensus.cc:2258] T dfb42459c86140f8bd9cd1b798e5a197 P fb09133e8ecc409ba9ffc6044c90180f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:12:16.322086 32358 ts_tablet_manager.cc:1905] T dfb42459c86140f8bd9cd1b798e5a197 P fb09133e8ecc409ba9ffc6044c90180f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:12:16.324386 32360 ts_tablet_manager.cc:1918] T dfb42459c86140f8bd9cd1b798e5a197 P ba716e8ff56743de8c3094b708c833a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 16:12:16.324549 32360 log.cc:1196] T dfb42459c86140f8bd9cd1b798e5a197 P ba716e8ff56743de8c3094b708c833a7: Deleting WAL directory at /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/minicluster-data/ts-2/wal/wals/dfb42459c86140f8bd9cd1b798e5a197
I20260924 16:12:16.325059 32360 ts_tablet_manager.cc:1939] T dfb42459c86140f8bd9cd1b798e5a197 P ba716e8ff56743de8c3094b708c833a7: Deleting consensus metadata
I20260924 16:12:16.326339 31766 catalog_manager.cc:4841] TS ba716e8ff56743de8c3094b708c833a7 (127.29.234.3:39383): tablet dfb42459c86140f8bd9cd1b798e5a197 (table test-table [id=37c4b5c5ec534d22997f524fd6ca8d93]) successfully deleted
I20260924 16:12:16.329559 32358 ts_tablet_manager.cc:1918] T dfb42459c86140f8bd9cd1b798e5a197 P fb09133e8ecc409ba9ffc6044c90180f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 16:12:16.329716 32358 log.cc:1196] T dfb42459c86140f8bd9cd1b798e5a197 P fb09133e8ecc409ba9ffc6044c90180f: Deleting WAL directory at /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790266323169505-30632-0/minicluster-data/ts-1/wal/wals/dfb42459c86140f8bd9cd1b798e5a197
I20260924 16:12:16.330199 32358 ts_tablet_manager.cc:1939] T dfb42459c86140f8bd9cd1b798e5a197 P fb09133e8ecc409ba9ffc6044c90180f: Deleting consensus metadata
I20260924 16:12:16.331576 31764 catalog_manager.cc:4841] TS fb09133e8ecc409ba9ffc6044c90180f (127.29.234.2:44825): tablet dfb42459c86140f8bd9cd1b798e5a197 (table test-table [id=37c4b5c5ec534d22997f524fd6ca8d93]) successfully deleted
Sep 24 16:12:16 dist-test-slave-v5j9 krb5kdc[31734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266335, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.29.234.1@KRBTEST.COM
I20260924 16:12:16.338878 32288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:16.308556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58508 (local address 127.29.234.1:35217)
0924 16:12:16.308738 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:16.308745 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:16.311005 (+ 2260us) server_negotiation.cc:407] Connection header received
0924 16:12:16.312662 (+ 1657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:16.312669 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:16.312772 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:16.312851 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:16.323175 (+ 10324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:16.324304 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:16.326620 (+ 2316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:16.327294 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:16.327405 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:16.332812 (+ 5407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:16.332827 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:16.332842 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:16.332880 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:16.336039 (+ 3159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:16.336793 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:16.336801 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:16.336803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:16.336930 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:16.337338 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:16.337406 (+ 68us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:16.337408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:16.337738 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:16.338062 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:16.338331 (+ 269us) server_negotiation.cc:299] Negotiation successful
0924 16:12:16.338559 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260924 16:12:16.339689 31984 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58508
I20260924 16:12:16.357059 32332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:16.342616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50746 (local address 127.29.234.62:39697)
0924 16:12:16.342909 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:16.342913 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:16.343005 (+ 92us) server_negotiation.cc:407] Connection header received
0924 16:12:16.343168 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:16.343172 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:16.343238 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:16.343317 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:16.344953 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:16.345833 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:16.347623 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:16.348126 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:16.348232 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:16.351046 (+ 2814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:16.351051 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:16.351054 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:16.351087 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:16.353909 (+ 2822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:16.354743 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:16.354746 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:16.354748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:16.354870 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:16.355300 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:16.355303 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:16.355305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:16.355570 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:16.355793 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:16.356298 (+ 505us) server_negotiation.cc:299] Negotiation successful
0924 16:12:16.356546 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143}
W20260924 16:12:16.357801 31778 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50746
I20260924 16:12:16.361079 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 31851
I20260924 16:12:16.375628 32288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:16.362317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58518 (local address 127.29.234.1:35217)
0924 16:12:16.362540 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:16.362545 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:16.362562 (+ 17us) server_negotiation.cc:407] Connection header received
0924 16:12:16.362621 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:16.362625 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:16.362681 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:16.362784 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:16.364237 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:16.364981 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:16.366346 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:16.366845 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:16.366981 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:16.369831 (+ 2850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:16.369845 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:16.369848 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:16.369883 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:16.372863 (+ 2980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:16.373664 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:16.373672 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:16.373675 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:16.373816 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:16.374223 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:16.374228 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:16.374231 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:16.374501 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:16.374796 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:16.375065 (+ 269us) server_negotiation.cc:299] Negotiation successful
0924 16:12:16.375285 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260924 16:12:16.376339 31984 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58518
W20260924 16:12:16.376965 30632 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 31851: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 16:12:16.377228 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 31851
I20260924 16:12:16.396333 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 31993
Sep 24 16:12:16 dist-test-slave-v5j9 krb5kdc[31734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266335, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.29.234.2@KRBTEST.COM
I20260924 16:12:16.412784 32340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:16.397145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50396 (local address 127.29.234.2:44825)
0924 16:12:16.397373 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:16.397379 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:16.397400 (+ 21us) server_negotiation.cc:407] Connection header received
0924 16:12:16.397600 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:16.397606 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:16.397699 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:16.397805 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:16.398908 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:16.400026 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:16.401230 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:16.401921 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:16.402062 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:16.406128 (+ 4066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:16.406163 (+ 35us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:16.406182 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:16.406232 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:16.409928 (+ 3696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:16.410577 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:16.410589 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:16.410594 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:16.410774 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:16.411135 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:16.411140 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:16.411144 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:16.411616 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:16.411950 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:16.412135 (+ 185us) server_negotiation.cc:299] Negotiation successful
0924 16:12:16.412372 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260924 16:12:16.413599 32126 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50396
W20260924 16:12:16.414284 30632 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 31993: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 16:12:16.414501 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 31993
I20260924 16:12:16.444835 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 32135
Sep 24 16:12:16 dist-test-slave-v5j9 krb5kdc[31734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266335, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.29.234.3@KRBTEST.COM
I20260924 16:12:16.460785 32291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:16.445607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42654 (local address 127.29.234.3:39383)
0924 16:12:16.445788 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:16.445794 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:16.445945 (+ 151us) server_negotiation.cc:407] Connection header received
0924 16:12:16.446171 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:16.446175 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:16.446251 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:16.446380 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:16.447677 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:16.448470 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:16.449847 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:16.450257 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:16.450394 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:16.454499 (+ 4105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:16.454511 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:16.454528 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:16.454560 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:16.457933 (+ 3373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:16.458694 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:16.458703 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:16.458706 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:16.458852 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:16.459229 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:16.459234 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:16.459238 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:16.459700 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:16.460043 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:16.460242 (+ 199us) server_negotiation.cc:299] Negotiation successful
0924 16:12:16.460455 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
W20260924 16:12:16.461539 32268 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42654
W20260924 16:12:16.462215 30632 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 32135: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 16:12:16.462389 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 32135
I20260924 16:12:16.483261 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 31747
I20260924 16:12:16.484490 31820 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:12:16.600450 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 31747
2026-09-24T16:12:16Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4808 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:12:16 dist-test-slave-v5j9 krb5kdc[32370](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:12:16 dist-test-slave-v5j9 krb5kdc[32370](info): set up 2 sockets
Sep 24 16:12:16 dist-test-slave-v5j9 krb5kdc[32370](info): commencing operation
krb5kdc: starting...
W20260924 16:12:17.696933 30632 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Sep 24 16:12:17 dist-test-slave-v5j9 krb5kdc[32370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:12:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:12:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.234.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab.
Entry for principal kudu/127.29.234.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab.
I20260924 16:12:17.890545 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:39565
--webserver_interface=127.29.234.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab
--principal=kudu/127.29.234.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:37659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.234.62:39565
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:12:18.216270 32383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:18.216665 32383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:18.216841 32383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:18.224004 32383 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:12:18.224112 32383 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 16:12:18.224195 32383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:18.224290 32383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:18.224390 32383 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:12:18.224499 32383 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:12:18.234532 32383 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:37659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.234.62:39565
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab
--principal=kudu/127.29.234.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:39565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.234.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:12:18.235900 32383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:18.237392 32383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:18.250321 32388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:18.250293 32391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:18.250357 32389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:18.251209 32383 server_base.cc:1034] running on GCE node
I20260924 16:12:18.252092 32383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:18.253665 32383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:18.254917 32383 hybrid_clock.cc:648] HybridClock initialized: now 1790266338254860 us; error 38 us; skew 500 ppm
I20260924 16:12:18.255252 32383 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:12:18 dist-test-slave-v5j9 krb5kdc[32370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266338, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:18.258484 32383 init.cc:377] Logged in from keytab as kudu/127.29.234.62@KRBTEST.COM (short username kudu)
I20260924 16:12:18.259995 32383 webserver.cc:460] Webserver started at http://127.29.234.62:34859/ using document root <none> and password file <none>
I20260924 16:12:18.260754 32383 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:18.260890 32383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:18.261215 32383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:18.264313 32383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/minicluster-data/master-0/data/instance:
uuid: "3838eedbbb89456096bd20aa4891ea73"
format_stamp: "Formatted at 2026-09-24 16:12:18 on dist-test-slave-v5j9"
server_key: "dafbea375df3a8638b39a22c2e7399fd"
server_key_iv: "b1bbb80b5acf3abd399e988e11bd23a0"
server_key_version: "encryptionkey@0"
I20260924 16:12:18.265130 32383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/minicluster-data/master-0/wal/instance:
uuid: "3838eedbbb89456096bd20aa4891ea73"
format_stamp: "Formatted at 2026-09-24 16:12:18 on dist-test-slave-v5j9"
server_key: "dafbea375df3a8638b39a22c2e7399fd"
server_key_iv: "b1bbb80b5acf3abd399e988e11bd23a0"
server_key_version: "encryptionkey@0"
I20260924 16:12:18.270700 32383 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 16:12:18.274386 32398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:18.275702 32383 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:12:18.275892 32383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/minicluster-data/master-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/minicluster-data/master-0/wal
uuid: "3838eedbbb89456096bd20aa4891ea73"
format_stamp: "Formatted at 2026-09-24 16:12:18 on dist-test-slave-v5j9"
server_key: "dafbea375df3a8638b39a22c2e7399fd"
server_key_iv: "b1bbb80b5acf3abd399e988e11bd23a0"
server_key_version: "encryptionkey@0"
I20260924 16:12:18.276113 32383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:18.291251 32383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:18.295589 32383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:18.295866 32383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:18.319458 32383 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.62:39565
I20260924 16:12:18.319478 32460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.62:39565 every 8 connection(s)
I20260924 16:12:18.320966 32383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
I20260924 16:12:18.324864 32461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:18.328840 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 32383
I20260924 16:12:18.329069 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/minicluster-data/master-0/wal/instance
I20260924 16:12:18.329926 30632 external_mini_cluster.cc:1428] Setting key f0d1c01d77d98249a11388060459b3d7
I20260924 16:12:18.338738 32461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3838eedbbb89456096bd20aa4891ea73: Bootstrap starting.
I20260924 16:12:18.342674 32461 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3838eedbbb89456096bd20aa4891ea73: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:18.344094 32461 log.cc:824] T 00000000000000000000000000000000 P 3838eedbbb89456096bd20aa4891ea73: Log is configured to *not* fsync() on all Append() calls
Sep 24 16:12:18 dist-test-slave-v5j9 krb5kdc[32370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:12:18.347537 32461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3838eedbbb89456096bd20aa4891ea73: No bootstrap required, opened a new log
I20260924 16:12:18.353754 32464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:18.332063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58054 (local address 127.29.234.62:39565)
0924 16:12:18.332843 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:18.332855 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:18.332898 (+ 43us) server_negotiation.cc:407] Connection header received
0924 16:12:18.334553 (+ 1655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:18.334577 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:18.334990 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:18.335325 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:18.337093 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:18.338574 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:18.340308 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:18.340994 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:18.341125 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:18.346424 (+ 5299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:18.346432 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:18.346447 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:18.346482 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:18.349640 (+ 3158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:18.350434 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:18.350440 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:18.350446 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:18.350606 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:18.350974 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:18.350977 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:18.350978 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:18.351502 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:18.351717 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:18.352170 (+ 453us) server_negotiation.cc:299] Negotiation successful
0924 16:12:18.352625 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":377,"threads_started":1}
I20260924 16:12:18.356428 32461 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3838eedbbb89456096bd20aa4891ea73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3838eedbbb89456096bd20aa4891ea73" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 39565 } }
I20260924 16:12:18.356839 32461 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3838eedbbb89456096bd20aa4891ea73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:18.356951 32461 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3838eedbbb89456096bd20aa4891ea73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3838eedbbb89456096bd20aa4891ea73, State: Initialized, Role: FOLLOWER
I20260924 16:12:18.357571 32461 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3838eedbbb89456096bd20aa4891ea73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3838eedbbb89456096bd20aa4891ea73" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 39565 } }
I20260924 16:12:18.357825 32461 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3838eedbbb89456096bd20aa4891ea73 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:12:18.357964 32461 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3838eedbbb89456096bd20aa4891ea73 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:12:18.358139 32461 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3838eedbbb89456096bd20aa4891ea73 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:18.360342 32461 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3838eedbbb89456096bd20aa4891ea73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3838eedbbb89456096bd20aa4891ea73" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 39565 } }
I20260924 16:12:18.360874 32461 leader_election.cc:302] T 00000000000000000000000000000000 P 3838eedbbb89456096bd20aa4891ea73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3838eedbbb89456096bd20aa4891ea73; no voters:
I20260924 16:12:18.361562 32461 leader_election.cc:288] T 00000000000000000000000000000000 P 3838eedbbb89456096bd20aa4891ea73 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:12:18.361680 32466 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3838eedbbb89456096bd20aa4891ea73 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:12:18.362949 32466 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3838eedbbb89456096bd20aa4891ea73 [term 1 LEADER]: Becoming Leader. State: Replica: 3838eedbbb89456096bd20aa4891ea73, State: Running, Role: LEADER
I20260924 16:12:18.363526 32466 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3838eedbbb89456096bd20aa4891ea73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3838eedbbb89456096bd20aa4891ea73" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 39565 } }
I20260924 16:12:18.363830 32461 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3838eedbbb89456096bd20aa4891ea73 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:12:18.366572 32468 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3838eedbbb89456096bd20aa4891ea73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3838eedbbb89456096bd20aa4891ea73. Latest consensus state: current_term: 1 leader_uuid: "3838eedbbb89456096bd20aa4891ea73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3838eedbbb89456096bd20aa4891ea73" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 39565 } } }
I20260924 16:12:18.366520 32467 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3838eedbbb89456096bd20aa4891ea73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3838eedbbb89456096bd20aa4891ea73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3838eedbbb89456096bd20aa4891ea73" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 39565 } } }
I20260924 16:12:18.366961 32467 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3838eedbbb89456096bd20aa4891ea73 [sys.catalog]: This master's current role is: LEADER
I20260924 16:12:18.366961 32468 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3838eedbbb89456096bd20aa4891ea73 [sys.catalog]: This master's current role is: LEADER
I20260924 16:12:18.369155 32474 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:12:18.374557 32474 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:12:18.383476 32474 catalog_manager.cc:1347] Generated new cluster ID: 930b281ddc3a4dd5a5ec9bc24f43df1e
I20260924 16:12:18.383607 32474 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:12:18.397159 32474 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:12:18.398511 32474 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:12:18.421643 32474 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3838eedbbb89456096bd20aa4891ea73: Generated new TSK 0
I20260924 16:12:18.422935 32474 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.234.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab.
Entry for principal kudu/127.29.234.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab.
I20260924 16:12:18.476784 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.1:0
--local_ip_for_outbound_sockets=127.29.234.1
--webserver_interface=127.29.234.1
--webserver_port=0
--tserver_master_addrs=127.29.234.62:39565
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab
--principal=kudu/127.29.234.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:37659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:12:18.811084 32487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:18.811375 32487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:18.811475 32487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:18.818899 32487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:18.819015 32487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:18.819099 32487 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 16:12:18.819151 32487 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 16:12:18.819212 32487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.1
I20260924 16:12:18.829008 32487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:37659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab
--principal=kudu/127.29.234.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.234.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.234.62:39565
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.1
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:12:18.830353 32487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:18.831643 32487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:18.845208 32492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:18.845080 32493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:18.845131 32496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:18.846000 32487 server_base.cc:1034] running on GCE node
I20260924 16:12:18.846721 32487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:18.847741 32487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:18.849001 32487 hybrid_clock.cc:648] HybridClock initialized: now 1790266338848927 us; error 63 us; skew 500 ppm
I20260924 16:12:18.849440 32487 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:12:18 dist-test-slave-v5j9 krb5kdc[32370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266338, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:18.852753 32487 init.cc:377] Logged in from keytab as kudu/127.29.234.1@KRBTEST.COM (short username kudu)
I20260924 16:12:18.854300 32487 webserver.cc:460] Webserver started at http://127.29.234.1:33965/ using document root <none> and password file <none>
I20260924 16:12:18.854997 32487 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:18.855180 32487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:18.855515 32487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:18.858536 32487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/minicluster-data/ts-0/data/instance:
uuid: "445ada5dbd2542b7a204077a5cf1471d"
format_stamp: "Formatted at 2026-09-24 16:12:18 on dist-test-slave-v5j9"
server_key: "bb2f10dba0076180d24f45ee0b88ee3a"
server_key_iv: "60327350bcdf4f9c037ddd848eb74f57"
server_key_version: "encryptionkey@0"
I20260924 16:12:18.859308 32487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/minicluster-data/ts-0/wal/instance:
uuid: "445ada5dbd2542b7a204077a5cf1471d"
format_stamp: "Formatted at 2026-09-24 16:12:18 on dist-test-slave-v5j9"
server_key: "bb2f10dba0076180d24f45ee0b88ee3a"
server_key_iv: "60327350bcdf4f9c037ddd848eb74f57"
server_key_version: "encryptionkey@0"
I20260924 16:12:18.864670 32487 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260924 16:12:18.868268 32503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:18.869762 32487 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:12:18.869938 32487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/minicluster-data/ts-0/wal
uuid: "445ada5dbd2542b7a204077a5cf1471d"
format_stamp: "Formatted at 2026-09-24 16:12:18 on dist-test-slave-v5j9"
server_key: "bb2f10dba0076180d24f45ee0b88ee3a"
server_key_iv: "60327350bcdf4f9c037ddd848eb74f57"
server_key_version: "encryptionkey@0"
I20260924 16:12:18.870138 32487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:18.887276 32487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:18.891368 32487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:18.891664 32487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:18.894677 32487 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:12:18.894800 32487 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:18.894920 32487 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:12:18.895016 32487 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 16:12:18 dist-test-slave-v5j9 krb5kdc[32370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266338, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.1@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:12:18.918220 32511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:18.894917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.1:60829 (local address 127.29.234.62:39565)
0924 16:12:18.895505 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:18.895512 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:18.896862 (+ 1350us) server_negotiation.cc:407] Connection header received
0924 16:12:18.898970 (+ 2108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:18.898980 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:18.899091 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:18.899215 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:18.901515 (+ 2300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:18.902572 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:18.905265 (+ 2693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:18.905819 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:18.905984 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:18.911021 (+ 5037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:18.911030 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:18.911039 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:18.911072 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:18.913696 (+ 2624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:18.914932 (+ 1236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:18.914939 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:18.914943 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:18.915168 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:18.915849 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:18.915855 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:18.915858 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:18.916306 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:18.916489 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:18.917575 (+ 1086us) server_negotiation.cc:299] Negotiation successful
0924 16:12:18.917821 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":284,"threads_started":1}
I20260924 16:12:18.919410 32513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:18.895589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:39565 (local address 127.29.234.1:60829)
0924 16:12:18.896637 (+ 1048us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:18.896722 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:18.898680 (+ 1958us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:18.899483 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:18.899493 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:18.900316 (+ 823us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:18.901275 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:18.901312 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:18.902734 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:18.902746 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:18.905117 (+ 2371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:18.905130 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:18.906120 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:18.906130 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:18.906571 (+ 441us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:18.907399 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:18.907436 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:18.910794 (+ 3358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:18.913887 (+ 3093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:18.913897 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:18.913920 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:18.914759 (+ 839us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:18.915313 (+ 554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:18.915320 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:18.915326 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:18.915630 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:18.916489 (+ 859us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:18.916504 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:18.917063 (+ 559us) client_negotiation.cc:769] Sending connection context
0924 16:12:18.917482 (+ 419us) client_negotiation.cc:240] Negotiation successful
0924 16:12:18.918078 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":756,"thread_start_us":401,"threads_started":1}
I20260924 16:12:18.942463 32487 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.1:46803
I20260924 16:12:18.942559 32626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.1:46803 every 8 connection(s)
I20260924 16:12:18.943993 32487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
I20260924 16:12:18.946035 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 32487
I20260924 16:12:18.946265 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/minicluster-data/ts-0/wal/instance
I20260924 16:12:18.946839 32511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:18.925619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.1:33315 (local address 127.29.234.62:39565)
0924 16:12:18.925853 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:18.925859 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:18.930392 (+ 4533us) server_negotiation.cc:407] Connection header received
0924 16:12:18.930691 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:18.930696 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:18.930776 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:18.930902 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:18.932747 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:18.933769 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:18.936299 (+ 2530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:18.937022 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:18.937184 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:18.940076 (+ 2892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:18.940085 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:18.940091 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:18.940138 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:18.943341 (+ 3203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:18.944229 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:18.944233 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:18.944235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:18.944361 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:18.944878 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:18.944887 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:18.944892 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:18.945550 (+ 658us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:18.945677 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:18.946263 (+ 586us) server_negotiation.cc:299] Negotiation successful
0924 16:12:18.946487 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260924 16:12:18.946920 32579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:18.925826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:39565 (local address 127.29.234.1:33315)
0924 16:12:18.930248 (+ 4422us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:18.930283 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:18.930537 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:18.930960 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:18.930967 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:18.931831 (+ 864us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:18.932528 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:18.932557 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:18.933930 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:18.933938 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:18.936054 (+ 2116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:18.936080 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:18.937291 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:18.937300 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:18.937563 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:18.938355 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:18.938383 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:18.939852 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:18.943472 (+ 3620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:18.943477 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:18.943481 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:18.944052 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:18.944486 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:18.944494 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:18.944500 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:18.944696 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:18.945783 (+ 1087us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:18.945788 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:18.945988 (+ 200us) client_negotiation.cc:769] Sending connection context
0924 16:12:18.946159 (+ 171us) client_negotiation.cc:240] Negotiation successful
0924 16:12:18.946482 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4194,"thread_start_us":3833,"threads_started":1}
I20260924 16:12:18.946977 30632 external_mini_cluster.cc:1428] Setting key 91053af18a2d4baaf8656fc421a2c410
I20260924 16:12:18.947964 32627 heartbeater.cc:347] Connected to a master server at 127.29.234.62:39565
I20260924 16:12:18.948211 32627 heartbeater.cc:464] Registering TS with master...
I20260924 16:12:18.949079 32627 heartbeater.cc:511] Master 127.29.234.62:39565 requested a full tablet report, sending...
I20260924 16:12:18.951480 32415 ts_manager.cc:194] Registered new tserver with Master: 445ada5dbd2542b7a204077a5cf1471d (127.29.234.1:46803)
I20260924 16:12:18.953598 32415 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.234.1@KRBTEST.COM'} at 127.29.234.1:60829
WARNING: no policy specified for kudu/127.29.234.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab.
Entry for principal kudu/127.29.234.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab.
I20260924 16:12:18.990684 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.2:0
--local_ip_for_outbound_sockets=127.29.234.2
--webserver_interface=127.29.234.2
--webserver_port=0
--tserver_master_addrs=127.29.234.62:39565
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab
--principal=kudu/127.29.234.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:37659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:12:19.315017 32631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:19.315382 32631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:19.315549 32631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:19.325330 32631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:19.325490 32631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:19.325628 32631 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 16:12:19.325704 32631 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 16:12:19.325817 32631 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.2
I20260924 16:12:19.338037 32631 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:37659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab
--principal=kudu/127.29.234.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.234.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.234.62:39565
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.2
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:12:19.339288 32631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:19.340608 32631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:19.354382 32636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:19.354452 32637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:19.354416 32639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:19.355351 32631 server_base.cc:1034] running on GCE node
I20260924 16:12:19.355908 32631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:19.356796 32631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:19.358011 32631 hybrid_clock.cc:648] HybridClock initialized: now 1790266339357957 us; error 39 us; skew 500 ppm
I20260924 16:12:19.358333 32631 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:12:19 dist-test-slave-v5j9 krb5kdc[32370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266339, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:19.361400 32631 init.cc:377] Logged in from keytab as kudu/127.29.234.2@KRBTEST.COM (short username kudu)
I20260924 16:12:19.362811 32631 webserver.cc:460] Webserver started at http://127.29.234.2:40339/ using document root <none> and password file <none>
I20260924 16:12:19.363466 32631 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:19.363613 32631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:19.363965 32631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:19.366981 32631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/minicluster-data/ts-1/data/instance:
uuid: "9d2bdbac60244a5d9a356958db1bb1c7"
format_stamp: "Formatted at 2026-09-24 16:12:19 on dist-test-slave-v5j9"
server_key: "0079f359a9d80bdaa97f69653a76e67e"
server_key_iv: "943ca2608463e79fd4fde703e5511cf1"
server_key_version: "encryptionkey@0"
I20260924 16:12:19.367827 32631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/minicluster-data/ts-1/wal/instance:
uuid: "9d2bdbac60244a5d9a356958db1bb1c7"
format_stamp: "Formatted at 2026-09-24 16:12:19 on dist-test-slave-v5j9"
server_key: "0079f359a9d80bdaa97f69653a76e67e"
server_key_iv: "943ca2608463e79fd4fde703e5511cf1"
server_key_version: "encryptionkey@0"
I20260924 16:12:19.373158 32631 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 16:12:19.377082 32646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:19.378487 32631 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:12:19.378671 32631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/minicluster-data/ts-1/wal
uuid: "9d2bdbac60244a5d9a356958db1bb1c7"
format_stamp: "Formatted at 2026-09-24 16:12:19 on dist-test-slave-v5j9"
server_key: "0079f359a9d80bdaa97f69653a76e67e"
server_key_iv: "943ca2608463e79fd4fde703e5511cf1"
server_key_version: "encryptionkey@0"
I20260924 16:12:19.378888 32631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:19.397634 32631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:19.401708 32631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:19.402005 32631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:19.404621 32631 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:12:19.404752 32631 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:19.404903 32631 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:12:19.405004 32631 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 16:12:19 dist-test-slave-v5j9 krb5kdc[32370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266339, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.2@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:12:19.432576 32655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:19.406032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:39565 (local address 127.29.234.2:59587)
0924 16:12:19.406948 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:19.407012 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:19.408898 (+ 1886us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:19.409916 (+ 1018us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:19.409930 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:19.410970 (+ 1040us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:19.412084 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:19.412124 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:19.413616 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:19.413621 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:19.416561 (+ 2940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:19.416578 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:19.417918 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:19.417930 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:19.418281 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:19.419191 (+ 910us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:19.419217 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:19.422538 (+ 3321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:19.426924 (+ 4386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:19.426937 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:19.426963 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:19.427762 (+ 799us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:19.428220 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:19.428232 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:19.428235 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:19.428460 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:19.429163 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:19.429175 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:19.429781 (+ 606us) client_negotiation.cc:769] Sending connection context
0924 16:12:19.430234 (+ 453us) client_negotiation.cc:240] Negotiation successful
0924 16:12:19.430961 (+ 727us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"thread_start_us":385,"threads_started":1}
I20260924 16:12:19.432687 32511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:19.405481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.2:59587 (local address 127.29.234.62:39565)
0924 16:12:19.405702 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:19.405708 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:19.407160 (+ 1452us) server_negotiation.cc:407] Connection header received
0924 16:12:19.409204 (+ 2044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:19.409209 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:19.409298 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:19.409434 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:19.412314 (+ 2880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:19.413415 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:19.416816 (+ 3401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:19.417441 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:19.417578 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:19.422790 (+ 5212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:19.422795 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:19.422798 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:19.422832 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:19.426711 (+ 3879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:19.427913 (+ 1202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:19.427921 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:19.427926 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:19.428075 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:19.428582 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:19.428589 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:19.428592 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:19.428987 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:19.429135 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:19.431902 (+ 2767us) server_negotiation.cc:299] Negotiation successful
0924 16:12:19.432208 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260924 16:12:19.447857 32631 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.2:41705
I20260924 16:12:19.447937 32767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.2:41705 every 8 connection(s)
I20260924 16:12:19.449201 32631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
I20260924 16:12:19.451725 300 heartbeater.cc:347] Connected to a master server at 127.29.234.62:39565
I20260924 16:12:19.451977 300 heartbeater.cc:464] Registering TS with master...
I20260924 16:12:19.452767 300 heartbeater.cc:511] Master 127.29.234.62:39565 requested a full tablet report, sending...
I20260924 16:12:19.454326 32415 ts_manager.cc:194] Registered new tserver with Master: 9d2bdbac60244a5d9a356958db1bb1c7 (127.29.234.2:41705)
I20260924 16:12:19.454607 32511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:19.439810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.2:56397 (local address 127.29.234.62:39565)
0924 16:12:19.440051 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:19.440061 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:19.441094 (+ 1033us) server_negotiation.cc:407] Connection header received
0924 16:12:19.441415 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:19.441421 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:19.441495 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:19.441602 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:19.443053 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:19.444062 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:19.446015 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:19.446428 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:19.446570 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:19.448693 (+ 2123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:19.448704 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:19.448708 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:19.448737 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:19.451921 (+ 3184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:19.452605 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:19.452612 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:19.452615 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:19.452756 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:19.453121 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:19.453127 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:19.453130 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:19.453529 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:19.453698 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:19.454109 (+ 411us) server_negotiation.cc:299] Negotiation successful
0924 16:12:19.454270 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 16:12:19.454715 32749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:19.439818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:39565 (local address 127.29.234.2:56397)
0924 16:12:19.440967 (+ 1149us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:19.440995 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:19.441186 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:19.441630 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:19.441635 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:19.442383 (+ 748us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:19.442838 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:19.442855 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:19.444245 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:19.444248 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:19.445869 (+ 1621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:19.445881 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:19.446632 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:19.446639 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:19.446827 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:19.447563 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:19.447594 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:19.448498 (+ 904us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:19.452046 (+ 3548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:19.452051 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:19.452061 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:19.452466 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:19.452878 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:19.452883 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:19.452887 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:19.453013 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:19.453657 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:19.453663 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:19.453847 (+ 184us) client_negotiation.cc:769] Sending connection context
0924 16:12:19.454015 (+ 168us) client_negotiation.cc:240] Negotiation successful
0924 16:12:19.454332 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":996,"thread_start_us":846,"threads_started":1}
I20260924 16:12:19.455386 32415 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.234.2@KRBTEST.COM'} at 127.29.234.2:59587
I20260924 16:12:19.458921 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 32631
I20260924 16:12:19.459105 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/minicluster-data/ts-1/wal/instance
I20260924 16:12:19.459537 30632 external_mini_cluster.cc:1428] Setting key 2a53d97383f221f08355434f105ccc54
WARNING: no policy specified for kudu/127.29.234.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab.
Entry for principal kudu/127.29.234.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab.
I20260924 16:12:19.500231 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.3:0
--local_ip_for_outbound_sockets=127.29.234.3
--webserver_interface=127.29.234.3
--webserver_port=0
--tserver_master_addrs=127.29.234.62:39565
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab
--principal=kudu/127.29.234.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:37659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:12:19.824896 304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:19.825160 304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:19.825335 304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:19.832532 304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:19.832651 304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:19.832801 304 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 16:12:19.832887 304 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 16:12:19.833029 304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.3
I20260924 16:12:19.846370 304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:37659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab
--principal=kudu/127.29.234.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.234.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.234.62:39565
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.3
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:12:19.847734 304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:19.849079 304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:19.863386 309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:19.863313 310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:19.863363 312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:19.866681 304 server_base.cc:1034] running on GCE node
I20260924 16:12:19.867362 304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:19.868427 304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:19.869655 304 hybrid_clock.cc:648] HybridClock initialized: now 1790266339869612 us; error 33 us; skew 500 ppm
I20260924 16:12:19.869995 304 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:12:19 dist-test-slave-v5j9 krb5kdc[32370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266339, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:19.873123 304 init.cc:377] Logged in from keytab as kudu/127.29.234.3@KRBTEST.COM (short username kudu)
I20260924 16:12:19.874856 304 webserver.cc:460] Webserver started at http://127.29.234.3:41695/ using document root <none> and password file <none>
I20260924 16:12:19.875626 304 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:19.875784 304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:19.876129 304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:19.879406 304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/minicluster-data/ts-2/data/instance:
uuid: "6ae33e71c7d2437abcb65ed59923573d"
format_stamp: "Formatted at 2026-09-24 16:12:19 on dist-test-slave-v5j9"
server_key: "1f472907a45c418a0fb63668ace03639"
server_key_iv: "2f780bd6e63a69ee76bbab5efb9793a9"
server_key_version: "encryptionkey@0"
I20260924 16:12:19.880266 304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/minicluster-data/ts-2/wal/instance:
uuid: "6ae33e71c7d2437abcb65ed59923573d"
format_stamp: "Formatted at 2026-09-24 16:12:19 on dist-test-slave-v5j9"
server_key: "1f472907a45c418a0fb63668ace03639"
server_key_iv: "2f780bd6e63a69ee76bbab5efb9793a9"
server_key_version: "encryptionkey@0"
I20260924 16:12:19.885859 304 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 16:12:19.889240 319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:19.890571 304 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:12:19.890749 304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/minicluster-data/ts-2/wal
uuid: "6ae33e71c7d2437abcb65ed59923573d"
format_stamp: "Formatted at 2026-09-24 16:12:19 on dist-test-slave-v5j9"
server_key: "1f472907a45c418a0fb63668ace03639"
server_key_iv: "2f780bd6e63a69ee76bbab5efb9793a9"
server_key_version: "encryptionkey@0"
I20260924 16:12:19.890956 304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:19.920252 304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:19.924608 304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:19.924906 304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:19.927584 304 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:12:19.927713 304 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:19.927843 304 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:12:19.927940 304 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 16:12:19 dist-test-slave-v5j9 krb5kdc[32370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266339, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.3@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:12:19.952293 32511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:19.928432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.3:46837 (local address 127.29.234.62:39565)
0924 16:12:19.928707 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:19.928713 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:19.930095 (+ 1382us) server_negotiation.cc:407] Connection header received
0924 16:12:19.932169 (+ 2074us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:19.932174 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:19.932263 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:19.932382 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:19.935446 (+ 3064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:19.936478 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:19.938588 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:19.939483 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:19.939659 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:19.943738 (+ 4079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:19.943757 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:19.943766 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:19.943823 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:19.948169 (+ 4346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:19.948905 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:19.948913 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:19.948919 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:19.949102 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:19.949888 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:19.949895 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:19.949898 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:19.950214 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:19.950402 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:19.951493 (+ 1091us) server_negotiation.cc:299] Negotiation successful
0924 16:12:19.951804 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260924 16:12:19.953402 328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:19.928987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:39565 (local address 127.29.234.3:46837)
0924 16:12:19.929909 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:19.929967 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:19.931887 (+ 1920us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:19.932878 (+ 991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:19.932899 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:19.933946 (+ 1047us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:19.935214 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:19.935252 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:19.936637 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:19.936642 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:19.938415 (+ 1773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:19.938433 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:19.939709 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:19.939714 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:19.939908 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:19.940649 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:19.940670 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:19.943464 (+ 2794us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:19.948317 (+ 4853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:19.948329 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:19.948346 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:19.948756 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:19.949369 (+ 613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:19.949379 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:19.949384 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:19.949698 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:19.950510 (+ 812us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:19.950520 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:19.950899 (+ 379us) client_negotiation.cc:769] Sending connection context
0924 16:12:19.951274 (+ 375us) client_negotiation.cc:240] Negotiation successful
0924 16:12:19.951937 (+ 663us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":415,"threads_started":1}
I20260924 16:12:19.957087 32627 heartbeater.cc:503] Master 127.29.234.62:39565 was elected leader, sending a full tablet report...
I20260924 16:12:19.980840 32511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:19.965312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.3:56711 (local address 127.29.234.62:39565)
0924 16:12:19.965585 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:19.965594 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:19.966153 (+ 559us) server_negotiation.cc:407] Connection header received
0924 16:12:19.966320 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:19.966328 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:19.966447 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:19.966576 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:19.967480 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:19.968932 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:19.971126 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:19.971702 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:19.971836 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:19.974109 (+ 2273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:19.974119 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:19.974126 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:19.974179 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:19.977753 (+ 3574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:19.978685 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:19.978692 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:19.978697 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:19.978867 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:19.979282 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:19.979287 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:19.979293 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:19.979639 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:19.979818 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:19.980240 (+ 422us) server_negotiation.cc:299] Negotiation successful
0924 16:12:19.980478 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260924 16:12:19.980943 393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:19.965330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:39565 (local address 127.29.234.3:56711)
0924 16:12:19.966031 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:19.966049 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:19.966195 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:19.966609 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:19.966612 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:19.967055 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:19.967333 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:19.967342 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:19.969134 (+ 1792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:19.969142 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:19.970979 (+ 1837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:19.970996 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:19.971909 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:19.971921 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:19.972088 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:19.972807 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:19.972823 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:19.973895 (+ 1072us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:19.977898 (+ 4003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:19.977903 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:19.977907 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:19.978552 (+ 645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:19.979012 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:19.979018 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:19.979022 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:19.979156 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:19.979761 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:19.979765 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:19.979961 (+ 196us) client_negotiation.cc:769] Sending connection context
0924 16:12:19.980131 (+ 170us) client_negotiation.cc:240] Negotiation successful
0924 16:12:19.980501 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":356,"threads_started":1}
I20260924 16:12:19.992106 304 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.3:42009
I20260924 16:12:19.992157 441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.3:42009 every 8 connection(s)
I20260924 16:12:19.993484 304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
I20260924 16:12:19.995790 442 heartbeater.cc:347] Connected to a master server at 127.29.234.62:39565
I20260924 16:12:19.995998 442 heartbeater.cc:464] Registering TS with master...
I20260924 16:12:19.996603 442 heartbeater.cc:511] Master 127.29.234.62:39565 requested a full tablet report, sending...
I20260924 16:12:19.997967 32414 ts_manager.cc:194] Registered new tserver with Master: 6ae33e71c7d2437abcb65ed59923573d (127.29.234.3:42009)
I20260924 16:12:19.998791 32414 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.234.3@KRBTEST.COM'} at 127.29.234.3:46837
I20260924 16:12:20.000059 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 304
I20260924 16:12:20.000232 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/minicluster-data/ts-2/wal/instance
I20260924 16:12:20.000766 30632 external_mini_cluster.cc:1428] Setting key 356d032d8e766ba0259c1c4286ca1c13
I20260924 16:12:20.003917 30632 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 16:12:20 dist-test-slave-v5j9 krb5kdc[32370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 16:12:20 dist-test-slave-v5j9 krb5kdc[32370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:12:20.051967 32511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:20.037749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58058 (local address 127.29.234.62:39565)
0924 16:12:20.037996 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:20.038005 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:20.038280 (+ 275us) server_negotiation.cc:407] Connection header received
0924 16:12:20.038504 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:20.038510 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:20.038589 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:20.038699 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:20.040204 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:20.041050 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:20.042327 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:20.042745 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:20.042883 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:20.046807 (+ 3924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:20.046812 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:20.046815 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:20.046851 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:20.049336 (+ 2485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:20.050036 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:20.050041 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:20.050045 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:20.050179 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:20.050556 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:20.050563 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:20.050571 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:20.050866 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:20.051109 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:20.051394 (+ 285us) server_negotiation.cc:299] Negotiation successful
0924 16:12:20.051574 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260924 16:12:20.056291 32414 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58058:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:12:20.063115 32414 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:12:20.087602 457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:20.075331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.3:42009 (local address 127.0.0.1:37924)
0924 16:12:20.078109 (+ 2778us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:20.078168 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:20.078418 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:20.079568 (+ 1150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:20.079577 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:20.079612 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:12:20.080033 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:20.080054 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:20.082036 (+ 1982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:20.082045 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:20.084295 (+ 2250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:20.084313 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:20.086504 (+ 2191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:20.086513 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:20.086724 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:20.086790 (+ 66us) client_negotiation.cc:769] Sending connection context
0924 16:12:20.086926 (+ 136us) client_negotiation.cc:240] Negotiation successful
0924 16:12:20.087129 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2421,"mutex_wait_us":49,"thread_start_us":3417,"threads_started":1}
I20260924 16:12:20.088299 456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:20.075642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37924 (local address 127.29.234.3:42009)
0924 16:12:20.076929 (+ 1287us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:20.076942 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:20.078301 (+ 1359us) server_negotiation.cc:407] Connection header received
0924 16:12:20.078653 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:20.078663 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:20.078955 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:20.079159 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:12:20.080290 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:20.081906 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:20.084476 (+ 2570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:20.085983 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:20.086131 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:20.086392 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:20.087773 (+ 1381us) server_negotiation.cc:299] Negotiation successful
0924 16:12:20.087957 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1078,"thread_start_us":860,"threads_started":1}
I20260924 16:12:20.088320 460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:20.079702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:46803 (local address 127.0.0.1:39246)
0924 16:12:20.080680 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:20.080692 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:20.080840 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:20.081439 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:20.081444 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:20.081469 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:12:20.081813 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:20.081823 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:20.083711 (+ 1888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:20.083715 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:20.085673 (+ 1958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:20.085688 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:20.087465 (+ 1777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:20.087471 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:20.087661 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:20.087698 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 16:12:20.087804 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 16:12:20.087916 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":881,"mutex_wait_us":1,"thread_start_us":696,"threads_started":1}
I20260924 16:12:20.088985 459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:20.079431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39246 (local address 127.29.234.1:46803)
0924 16:12:20.079972 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:20.079980 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:20.080795 (+ 815us) server_negotiation.cc:407] Connection header received
0924 16:12:20.081010 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:20.081017 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:20.081225 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:20.081438 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:12:20.081997 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:20.083575 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:20.085841 (+ 2266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:20.087251 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:20.087390 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:20.087599 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:20.088570 (+ 971us) server_negotiation.cc:299] Negotiation successful
0924 16:12:20.088768 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":394,"threads_started":1}
I20260924 16:12:20.089964 458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:20.076094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43082 (local address 127.29.234.2:41705)
0924 16:12:20.077398 (+ 1304us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:20.077411 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:20.079177 (+ 1766us) server_negotiation.cc:407] Connection header received
0924 16:12:20.079520 (+ 343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:20.079528 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:20.079795 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:20.079993 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:12:20.081027 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:20.084587 (+ 3560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:20.086627 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:20.087954 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:20.088092 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:20.088280 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:20.089501 (+ 1221us) server_negotiation.cc:299] Negotiation successful
0924 16:12:20.089689 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1107,"thread_start_us":990,"threads_started":1}
I20260924 16:12:20.090004 455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:20.075921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.2:41705 (local address 127.0.0.1:43082)
0924 16:12:20.079068 (+ 3147us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:20.079087 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:20.079299 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:20.080232 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:20.080238 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:20.080280 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:12:20.080827 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:20.080843 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:20.084707 (+ 3864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:20.084715 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:20.086444 (+ 1729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:20.086459 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:20.088442 (+ 1983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:20.088450 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:20.088634 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:20.088667 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 16:12:20.088751 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 16:12:20.088873 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3012,"thread_start_us":1643,"threads_started":1}
I20260924 16:12:20.092456 369 tablet_service.cc:1467] Processing CreateTablet for tablet 6af0d9fd1c4d4dd58ce9d9b2d3637371 (DEFAULT_TABLE table=test-table [id=8d41c2a8d427470f804dad19c6181bf1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:12:20.092582 32555 tablet_service.cc:1467] Processing CreateTablet for tablet 6af0d9fd1c4d4dd58ce9d9b2d3637371 (DEFAULT_TABLE table=test-table [id=8d41c2a8d427470f804dad19c6181bf1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:12:20.093891 369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6af0d9fd1c4d4dd58ce9d9b2d3637371. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:20.093444 32697 tablet_service.cc:1467] Processing CreateTablet for tablet 6af0d9fd1c4d4dd58ce9d9b2d3637371 (DEFAULT_TABLE table=test-table [id=8d41c2a8d427470f804dad19c6181bf1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:12:20.094008 32555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6af0d9fd1c4d4dd58ce9d9b2d3637371. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:20.094655 32697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6af0d9fd1c4d4dd58ce9d9b2d3637371. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:20.106580 461 tablet_bootstrap.cc:492] T 6af0d9fd1c4d4dd58ce9d9b2d3637371 P 6ae33e71c7d2437abcb65ed59923573d: Bootstrap starting.
I20260924 16:12:20.108480 463 tablet_bootstrap.cc:492] T 6af0d9fd1c4d4dd58ce9d9b2d3637371 P 9d2bdbac60244a5d9a356958db1bb1c7: Bootstrap starting.
I20260924 16:12:20.109263 462 tablet_bootstrap.cc:492] T 6af0d9fd1c4d4dd58ce9d9b2d3637371 P 445ada5dbd2542b7a204077a5cf1471d: Bootstrap starting.
I20260924 16:12:20.110801 461 tablet_bootstrap.cc:654] T 6af0d9fd1c4d4dd58ce9d9b2d3637371 P 6ae33e71c7d2437abcb65ed59923573d: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:20.112068 463 tablet_bootstrap.cc:654] T 6af0d9fd1c4d4dd58ce9d9b2d3637371 P 9d2bdbac60244a5d9a356958db1bb1c7: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:20.112143 461 log.cc:824] T 6af0d9fd1c4d4dd58ce9d9b2d3637371 P 6ae33e71c7d2437abcb65ed59923573d: Log is configured to *not* fsync() on all Append() calls
I20260924 16:12:20.112742 462 tablet_bootstrap.cc:654] T 6af0d9fd1c4d4dd58ce9d9b2d3637371 P 445ada5dbd2542b7a204077a5cf1471d: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:20.113214 463 log.cc:824] T 6af0d9fd1c4d4dd58ce9d9b2d3637371 P 9d2bdbac60244a5d9a356958db1bb1c7: Log is configured to *not* fsync() on all Append() calls
I20260924 16:12:20.113955 462 log.cc:824] T 6af0d9fd1c4d4dd58ce9d9b2d3637371 P 445ada5dbd2542b7a204077a5cf1471d: Log is configured to *not* fsync() on all Append() calls
I20260924 16:12:20.115701 461 tablet_bootstrap.cc:492] T 6af0d9fd1c4d4dd58ce9d9b2d3637371 P 6ae33e71c7d2437abcb65ed59923573d: No bootstrap required, opened a new log
I20260924 16:12:20.115999 461 ts_tablet_manager.cc:1397] T 6af0d9fd1c4d4dd58ce9d9b2d3637371 P 6ae33e71c7d2437abcb65ed59923573d: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260924 16:12:20.116966 462 tablet_bootstrap.cc:492] T 6af0d9fd1c4d4dd58ce9d9b2d3637371 P 445ada5dbd2542b7a204077a5cf1471d: No bootstrap required, opened a new log
I20260924 16:12:20.117233 462 ts_tablet_manager.cc:1397] T 6af0d9fd1c4d4dd58ce9d9b2d3637371 P 445ada5dbd2542b7a204077a5cf1471d: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260924 16:12:20.117975 463 tablet_bootstrap.cc:492] T 6af0d9fd1c4d4dd58ce9d9b2d3637371 P 9d2bdbac60244a5d9a356958db1bb1c7: No bootstrap required, opened a new log
I20260924 16:12:20.118227 463 ts_tablet_manager.cc:1397] T 6af0d9fd1c4d4dd58ce9d9b2d3637371 P 9d2bdbac60244a5d9a356958db1bb1c7: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260924 16:12:20.124574 462 raft_consensus.cc:348] T 6af0d9fd1c4d4dd58ce9d9b2d3637371 P 445ada5dbd2542b7a204077a5cf1471d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae33e71c7d2437abcb65ed59923573d" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 42009 } } peers { permanent_uuid: "445ada5dbd2542b7a204077a5cf1471d" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 46803 } } peers { permanent_uuid: "9d2bdbac60244a5d9a356958db1bb1c7" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41705 } }
I20260924 16:12:20.125144 462 raft_consensus.cc:374] T 6af0d9fd1c4d4dd58ce9d9b2d3637371 P 445ada5dbd2542b7a204077a5cf1471d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:20.125301 462 raft_consensus.cc:729] T 6af0d9fd1c4d4dd58ce9d9b2d3637371 P 445ada5dbd2542b7a204077a5cf1471d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 445ada5dbd2542b7a204077a5cf1471d, State: Initialized, Role: FOLLOWER
I20260924 16:12:20.126001 462 consensus_queue.cc:260] T 6af0d9fd1c4d4dd58ce9d9b2d3637371 P 445ada5dbd2542b7a204077a5cf1471d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae33e71c7d2437abcb65ed59923573d" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 42009 } } peers { permanent_uuid: "445ada5dbd2542b7a204077a5cf1471d" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 46803 } } peers { permanent_uuid: "9d2bdbac60244a5d9a356958db1bb1c7" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41705 } }
I20260924 16:12:20.126879 461 raft_consensus.cc:348] T 6af0d9fd1c4d4dd58ce9d9b2d3637371 P 6ae33e71c7d2437abcb65ed59923573d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae33e71c7d2437abcb65ed59923573d" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 42009 } } peers { permanent_uuid: "445ada5dbd2542b7a204077a5cf1471d" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 46803 } } peers { permanent_uuid: "9d2bdbac60244a5d9a356958db1bb1c7" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41705 } }
I20260924 16:12:20.127333 461 raft_consensus.cc:374] T 6af0d9fd1c4d4dd58ce9d9b2d3637371 P 6ae33e71c7d2437abcb65ed59923573d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:20.127445 461 raft_consensus.cc:729] T 6af0d9fd1c4d4dd58ce9d9b2d3637371 P 6ae33e71c7d2437abcb65ed59923573d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ae33e71c7d2437abcb65ed59923573d, State: Initialized, Role: FOLLOWER
I20260924 16:12:20.127731 462 ts_tablet_manager.cc:1428] T 6af0d9fd1c4d4dd58ce9d9b2d3637371 P 445ada5dbd2542b7a204077a5cf1471d: Time spent starting tablet: real 0.010s user 0.002s sys 0.009s
I20260924 16:12:20.128005 461 consensus_queue.cc:260] T 6af0d9fd1c4d4dd58ce9d9b2d3637371 P 6ae33e71c7d2437abcb65ed59923573d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae33e71c7d2437abcb65ed59923573d" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 42009 } } peers { permanent_uuid: "445ada5dbd2542b7a204077a5cf1471d" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 46803 } } peers { permanent_uuid: "9d2bdbac60244a5d9a356958db1bb1c7" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41705 } }
I20260924 16:12:20.128748 463 raft_consensus.cc:348] T 6af0d9fd1c4d4dd58ce9d9b2d3637371 P 9d2bdbac60244a5d9a356958db1bb1c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae33e71c7d2437abcb65ed59923573d" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 42009 } } peers { permanent_uuid: "445ada5dbd2542b7a204077a5cf1471d" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 46803 } } peers { permanent_uuid: "9d2bdbac60244a5d9a356958db1bb1c7" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41705 } }
I20260924 16:12:20.129153 442 heartbeater.cc:503] Master 127.29.234.62:39565 was elected leader, sending a full tablet report...
I20260924 16:12:20.129191 463 raft_consensus.cc:374] T 6af0d9fd1c4d4dd58ce9d9b2d3637371 P 9d2bdbac60244a5d9a356958db1bb1c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:20.129415 463 raft_consensus.cc:729] T 6af0d9fd1c4d4dd58ce9d9b2d3637371 P 9d2bdbac60244a5d9a356958db1bb1c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d2bdbac60244a5d9a356958db1bb1c7, State: Initialized, Role: FOLLOWER
I20260924 16:12:20.129853 461 ts_tablet_manager.cc:1428] T 6af0d9fd1c4d4dd58ce9d9b2d3637371 P 6ae33e71c7d2437abcb65ed59923573d: Time spent starting tablet: real 0.014s user 0.010s sys 0.006s
I20260924 16:12:20.130121 463 consensus_queue.cc:260] T 6af0d9fd1c4d4dd58ce9d9b2d3637371 P 9d2bdbac60244a5d9a356958db1bb1c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae33e71c7d2437abcb65ed59923573d" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 42009 } } peers { permanent_uuid: "445ada5dbd2542b7a204077a5cf1471d" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 46803 } } peers { permanent_uuid: "9d2bdbac60244a5d9a356958db1bb1c7" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41705 } }
I20260924 16:12:20.131237 300 heartbeater.cc:503] Master 127.29.234.62:39565 was elected leader, sending a full tablet report...
I20260924 16:12:20.131846 463 ts_tablet_manager.cc:1428] T 6af0d9fd1c4d4dd58ce9d9b2d3637371 P 9d2bdbac60244a5d9a356958db1bb1c7: Time spent starting tablet: real 0.014s user 0.008s sys 0.005s
W20260924 16:12:20.196436 32628 tablet.cc:2369] T 6af0d9fd1c4d4dd58ce9d9b2d3637371 P 445ada5dbd2542b7a204077a5cf1471d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 16:12:20.201316 301 tablet.cc:2369] T 6af0d9fd1c4d4dd58ce9d9b2d3637371 P 9d2bdbac60244a5d9a356958db1bb1c7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 16:12:20.245155 443 tablet.cc:2369] T 6af0d9fd1c4d4dd58ce9d9b2d3637371 P 6ae33e71c7d2437abcb65ed59923573d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:12:20.316359 468 raft_consensus.cc:482] T 6af0d9fd1c4d4dd58ce9d9b2d3637371 P 6ae33e71c7d2437abcb65ed59923573d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:12:20.316767 468 raft_consensus.cc:504] T 6af0d9fd1c4d4dd58ce9d9b2d3637371 P 6ae33e71c7d2437abcb65ed59923573d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae33e71c7d2437abcb65ed59923573d" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 42009 } } peers { permanent_uuid: "445ada5dbd2542b7a204077a5cf1471d" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 46803 } } peers { permanent_uuid: "9d2bdbac60244a5d9a356958db1bb1c7" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41705 } }
I20260924 16:12:20.319242 468 leader_election.cc:288] T 6af0d9fd1c4d4dd58ce9d9b2d3637371 P 6ae33e71c7d2437abcb65ed59923573d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 445ada5dbd2542b7a204077a5cf1471d (127.29.234.1:46803), 9d2bdbac60244a5d9a356958db1bb1c7 (127.29.234.2:41705)
I20260924 16:12:20.326646 458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:20.320096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.3:48363 (local address 127.29.234.2:41705)
0924 16:12:20.320321 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:20.320327 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:20.320643 (+ 316us) server_negotiation.cc:407] Connection header received
0924 16:12:20.320810 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:20.320816 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:20.320907 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:20.320997 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:12:20.321557 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:20.322529 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:20.324536 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:20.325485 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:20.325616 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:20.325690 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:20.326244 (+ 554us) server_negotiation.cc:299] Negotiation successful
0924 16:12:20.326379 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260924 16:12:20.326665 470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:20.319571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.2:41705 (local address 127.29.234.3:48363)
0924 16:12:20.320514 (+ 943us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:20.320533 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:20.320676 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:20.321026 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:20.321031 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:20.321054 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:12:20.321402 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:20.321414 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:20.322670 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:20.322675 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:20.324385 (+ 1710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:20.324398 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:20.325873 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:20.325880 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:20.326057 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:20.326090 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 16:12:20.326196 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 16:12:20.326331 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":810,"mutex_wait_us":28}
I20260924 16:12:20.327430 459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:20.319817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.3:59975 (local address 127.29.234.1:46803)
0924 16:12:20.320005 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:20.320012 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:20.320823 (+ 811us) server_negotiation.cc:407] Connection header received
0924 16:12:20.320903 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:20.320908 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:20.321006 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:20.321083 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:12:20.321849 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:20.322913 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:20.325187 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:20.326366 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:20.326516 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:20.326581 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:20.327053 (+ 472us) server_negotiation.cc:299] Negotiation successful
0924 16:12:20.327191 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 16:12:20.327478 328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:20.319832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:46803 (local address 127.29.234.3:59975)
0924 16:12:20.320540 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:20.320556 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:20.320701 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:20.321412 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:20.321417 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:20.321451 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:12:20.321704 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:20.321715 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:20.323082 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:20.323088 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:20.325024 (+ 1936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:20.325038 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:20.326676 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:20.326682 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:20.326889 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:20.326916 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 16:12:20.327004 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 16:12:20.327134 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"mutex_wait_us":3,"thread_start_us":353,"threads_started":1}
I20260924 16:12:20.328117 32717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af0d9fd1c4d4dd58ce9d9b2d3637371" candidate_uuid: "6ae33e71c7d2437abcb65ed59923573d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d2bdbac60244a5d9a356958db1bb1c7" is_pre_election: true
I20260924 16:12:20.328303 32580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af0d9fd1c4d4dd58ce9d9b2d3637371" candidate_uuid: "6ae33e71c7d2437abcb65ed59923573d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "445ada5dbd2542b7a204077a5cf1471d" is_pre_election: true
I20260924 16:12:20.328735 32717 raft_consensus.cc:2454] T 6af0d9fd1c4d4dd58ce9d9b2d3637371 P 9d2bdbac60244a5d9a356958db1bb1c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ae33e71c7d2437abcb65ed59923573d in term 0.
I20260924 16:12:20.328825 32580 raft_consensus.cc:2454] T 6af0d9fd1c4d4dd58ce9d9b2d3637371 P 445ada5dbd2542b7a204077a5cf1471d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ae33e71c7d2437abcb65ed59923573d in term 0.
I20260924 16:12:20.329461 322 leader_election.cc:302] T 6af0d9fd1c4d4dd58ce9d9b2d3637371 P 6ae33e71c7d2437abcb65ed59923573d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ae33e71c7d2437abcb65ed59923573d, 9d2bdbac60244a5d9a356958db1bb1c7; no voters:
I20260924 16:12:20.329875 468 raft_consensus.cc:2781] T 6af0d9fd1c4d4dd58ce9d9b2d3637371 P 6ae33e71c7d2437abcb65ed59923573d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:12:20.330039 468 raft_consensus.cc:482] T 6af0d9fd1c4d4dd58ce9d9b2d3637371 P 6ae33e71c7d2437abcb65ed59923573d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:12:20.330142 468 raft_consensus.cc:3037] T 6af0d9fd1c4d4dd58ce9d9b2d3637371 P 6ae33e71c7d2437abcb65ed59923573d [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:20.332283 468 raft_consensus.cc:504] T 6af0d9fd1c4d4dd58ce9d9b2d3637371 P 6ae33e71c7d2437abcb65ed59923573d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae33e71c7d2437abcb65ed59923573d" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 42009 } } peers { permanent_uuid: "445ada5dbd2542b7a204077a5cf1471d" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 46803 } } peers { permanent_uuid: "9d2bdbac60244a5d9a356958db1bb1c7" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41705 } }
I20260924 16:12:20.332890 468 leader_election.cc:288] T 6af0d9fd1c4d4dd58ce9d9b2d3637371 P 6ae33e71c7d2437abcb65ed59923573d [CANDIDATE]: Term 1 election: Requested vote from peers 445ada5dbd2542b7a204077a5cf1471d (127.29.234.1:46803), 9d2bdbac60244a5d9a356958db1bb1c7 (127.29.234.2:41705)
I20260924 16:12:20.333312 32580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af0d9fd1c4d4dd58ce9d9b2d3637371" candidate_uuid: "6ae33e71c7d2437abcb65ed59923573d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "445ada5dbd2542b7a204077a5cf1471d"
I20260924 16:12:20.333420 32717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af0d9fd1c4d4dd58ce9d9b2d3637371" candidate_uuid: "6ae33e71c7d2437abcb65ed59923573d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d2bdbac60244a5d9a356958db1bb1c7"
I20260924 16:12:20.333585 32580 raft_consensus.cc:3037] T 6af0d9fd1c4d4dd58ce9d9b2d3637371 P 445ada5dbd2542b7a204077a5cf1471d [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:20.333607 32717 raft_consensus.cc:3037] T 6af0d9fd1c4d4dd58ce9d9b2d3637371 P 9d2bdbac60244a5d9a356958db1bb1c7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:20.336124 32580 raft_consensus.cc:2454] T 6af0d9fd1c4d4dd58ce9d9b2d3637371 P 445ada5dbd2542b7a204077a5cf1471d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ae33e71c7d2437abcb65ed59923573d in term 1.
I20260924 16:12:20.336524 321 leader_election.cc:302] T 6af0d9fd1c4d4dd58ce9d9b2d3637371 P 6ae33e71c7d2437abcb65ed59923573d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 445ada5dbd2542b7a204077a5cf1471d, 6ae33e71c7d2437abcb65ed59923573d; no voters:
I20260924 16:12:20.336701 32717 raft_consensus.cc:2454] T 6af0d9fd1c4d4dd58ce9d9b2d3637371 P 9d2bdbac60244a5d9a356958db1bb1c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ae33e71c7d2437abcb65ed59923573d in term 1.
I20260924 16:12:20.336843 468 raft_consensus.cc:2781] T 6af0d9fd1c4d4dd58ce9d9b2d3637371 P 6ae33e71c7d2437abcb65ed59923573d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:12:20.337335 468 raft_consensus.cc:686] T 6af0d9fd1c4d4dd58ce9d9b2d3637371 P 6ae33e71c7d2437abcb65ed59923573d [term 1 LEADER]: Becoming Leader. State: Replica: 6ae33e71c7d2437abcb65ed59923573d, State: Running, Role: LEADER
I20260924 16:12:20.337932 468 consensus_queue.cc:237] T 6af0d9fd1c4d4dd58ce9d9b2d3637371 P 6ae33e71c7d2437abcb65ed59923573d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae33e71c7d2437abcb65ed59923573d" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 42009 } } peers { permanent_uuid: "445ada5dbd2542b7a204077a5cf1471d" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 46803 } } peers { permanent_uuid: "9d2bdbac60244a5d9a356958db1bb1c7" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41705 } }
I20260924 16:12:20.342409 32413 catalog_manager.cc:5495] T 6af0d9fd1c4d4dd58ce9d9b2d3637371 P 6ae33e71c7d2437abcb65ed59923573d reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ae33e71c7d2437abcb65ed59923573d (127.29.234.3). New cstate: current_term: 1 leader_uuid: "6ae33e71c7d2437abcb65ed59923573d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ae33e71c7d2437abcb65ed59923573d" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 42009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "445ada5dbd2542b7a204077a5cf1471d" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 46803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d2bdbac60244a5d9a356958db1bb1c7" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41705 } health_report { overall_health: UNKNOWN } } }
Sep 24 16:12:20 dist-test-slave-v5j9 krb5kdc[32370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266338, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.62@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:12:20.408651 32511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:20.395779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58066 (local address 127.29.234.62:39565)
0924 16:12:20.395974 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:20.395978 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:20.396478 (+ 500us) server_negotiation.cc:407] Connection header received
0924 16:12:20.396685 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:20.396693 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:20.396750 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:20.396851 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:20.398111 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:20.398844 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:20.400060 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:20.400485 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:20.400610 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:20.403717 (+ 3107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:20.403721 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:20.403724 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:20.403750 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:20.406032 (+ 2282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:20.406700 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:20.406707 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:20.406710 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:20.406840 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:20.407191 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:20.407198 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:20.407201 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:20.407470 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:20.407619 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:20.408044 (+ 425us) server_negotiation.cc:299] Negotiation successful
0924 16:12:20.408356 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":123,"server-negotiator.queue_time_us":61}
I20260924 16:12:20.408668 478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:20.395822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:39565 (local address 127.0.0.1:58066)
0924 16:12:20.396375 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:20.396390 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:20.396557 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:20.396872 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:20.396876 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:20.397700 (+ 824us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:20.397969 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:20.397978 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:20.398971 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:20.398979 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:20.399922 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:20.399934 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:20.400669 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:20.400675 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:20.400825 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:20.401512 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:20.401530 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:20.403551 (+ 2021us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:20.406167 (+ 2616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:20.406181 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:20.406189 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:20.406583 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:20.406943 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:20.406949 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:20.406953 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:20.407090 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:20.407603 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:20.407612 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:20.407777 (+ 165us) client_negotiation.cc:769] Sending connection context
0924 16:12:20.407943 (+ 166us) client_negotiation.cc:240] Negotiation successful
0924 16:12:20.408227 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":314,"threads_started":1}
I20260924 16:12:20.413587 32413 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.29.234.62@KRBTEST.COM'} at 127.0.0.1:58066:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 16:12:20.414618 32413 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:12:20.420971 32697 tablet_service.cc:1467] Processing CreateTablet for tablet 6618a7d814ad4961a5533f57d816e60d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d9c53fb0b1de42229b272ee6cb3d26b6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 16:12:20.420971 32555 tablet_service.cc:1467] Processing CreateTablet for tablet 6618a7d814ad4961a5533f57d816e60d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d9c53fb0b1de42229b272ee6cb3d26b6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 16:12:20.421083 369 tablet_service.cc:1467] Processing CreateTablet for tablet 6618a7d814ad4961a5533f57d816e60d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d9c53fb0b1de42229b272ee6cb3d26b6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 16:12:20.421975 32697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6618a7d814ad4961a5533f57d816e60d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:20.421978 32555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6618a7d814ad4961a5533f57d816e60d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:20.421983 369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6618a7d814ad4961a5533f57d816e60d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:20.427402 461 tablet_bootstrap.cc:492] T 6618a7d814ad4961a5533f57d816e60d P 6ae33e71c7d2437abcb65ed59923573d: Bootstrap starting.
I20260924 16:12:20.429180 463 tablet_bootstrap.cc:492] T 6618a7d814ad4961a5533f57d816e60d P 9d2bdbac60244a5d9a356958db1bb1c7: Bootstrap starting.
I20260924 16:12:20.429199 461 tablet_bootstrap.cc:654] T 6618a7d814ad4961a5533f57d816e60d P 6ae33e71c7d2437abcb65ed59923573d: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:20.429610 462 tablet_bootstrap.cc:492] T 6618a7d814ad4961a5533f57d816e60d P 445ada5dbd2542b7a204077a5cf1471d: Bootstrap starting.
I20260924 16:12:20.431151 462 tablet_bootstrap.cc:654] T 6618a7d814ad4961a5533f57d816e60d P 445ada5dbd2542b7a204077a5cf1471d: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:20.431500 461 tablet_bootstrap.cc:492] T 6618a7d814ad4961a5533f57d816e60d P 6ae33e71c7d2437abcb65ed59923573d: No bootstrap required, opened a new log
I20260924 16:12:20.431632 461 ts_tablet_manager.cc:1397] T 6618a7d814ad4961a5533f57d816e60d P 6ae33e71c7d2437abcb65ed59923573d: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260924 16:12:20.431646 463 tablet_bootstrap.cc:654] T 6618a7d814ad4961a5533f57d816e60d P 9d2bdbac60244a5d9a356958db1bb1c7: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:20.432471 461 raft_consensus.cc:348] T 6618a7d814ad4961a5533f57d816e60d P 6ae33e71c7d2437abcb65ed59923573d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445ada5dbd2542b7a204077a5cf1471d" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 46803 } } peers { permanent_uuid: "9d2bdbac60244a5d9a356958db1bb1c7" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41705 } } peers { permanent_uuid: "6ae33e71c7d2437abcb65ed59923573d" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 42009 } }
I20260924 16:12:20.432723 461 raft_consensus.cc:374] T 6618a7d814ad4961a5533f57d816e60d P 6ae33e71c7d2437abcb65ed59923573d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:20.432829 461 raft_consensus.cc:729] T 6618a7d814ad4961a5533f57d816e60d P 6ae33e71c7d2437abcb65ed59923573d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ae33e71c7d2437abcb65ed59923573d, State: Initialized, Role: FOLLOWER
I20260924 16:12:20.433156 461 consensus_queue.cc:260] T 6618a7d814ad4961a5533f57d816e60d P 6ae33e71c7d2437abcb65ed59923573d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445ada5dbd2542b7a204077a5cf1471d" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 46803 } } peers { permanent_uuid: "9d2bdbac60244a5d9a356958db1bb1c7" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41705 } } peers { permanent_uuid: "6ae33e71c7d2437abcb65ed59923573d" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 42009 } }
I20260924 16:12:20.433597 468 tablet_replica.cc:439] T 6618a7d814ad4961a5533f57d816e60d P 6ae33e71c7d2437abcb65ed59923573d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445ada5dbd2542b7a204077a5cf1471d" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 46803 } } peers { permanent_uuid: "9d2bdbac60244a5d9a356958db1bb1c7" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41705 } } peers { permanent_uuid: "6ae33e71c7d2437abcb65ed59923573d" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 42009 } } }
I20260924 16:12:20.433904 468 tablet_replica.cc:442] T 6618a7d814ad4961a5533f57d816e60d P 6ae33e71c7d2437abcb65ed59923573d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:12:20.434000 462 tablet_bootstrap.cc:492] T 6618a7d814ad4961a5533f57d816e60d P 445ada5dbd2542b7a204077a5cf1471d: No bootstrap required, opened a new log
I20260924 16:12:20.434167 462 ts_tablet_manager.cc:1397] T 6618a7d814ad4961a5533f57d816e60d P 445ada5dbd2542b7a204077a5cf1471d: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20260924 16:12:20.434486 461 ts_tablet_manager.cc:1428] T 6618a7d814ad4961a5533f57d816e60d P 6ae33e71c7d2437abcb65ed59923573d: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260924 16:12:20.435073 462 raft_consensus.cc:348] T 6618a7d814ad4961a5533f57d816e60d P 445ada5dbd2542b7a204077a5cf1471d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445ada5dbd2542b7a204077a5cf1471d" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 46803 } } peers { permanent_uuid: "9d2bdbac60244a5d9a356958db1bb1c7" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41705 } } peers { permanent_uuid: "6ae33e71c7d2437abcb65ed59923573d" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 42009 } }
I20260924 16:12:20.435355 462 raft_consensus.cc:374] T 6618a7d814ad4961a5533f57d816e60d P 445ada5dbd2542b7a204077a5cf1471d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:20.435514 462 raft_consensus.cc:729] T 6618a7d814ad4961a5533f57d816e60d P 445ada5dbd2542b7a204077a5cf1471d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 445ada5dbd2542b7a204077a5cf1471d, State: Initialized, Role: FOLLOWER
I20260924 16:12:20.435897 462 consensus_queue.cc:260] T 6618a7d814ad4961a5533f57d816e60d P 445ada5dbd2542b7a204077a5cf1471d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445ada5dbd2542b7a204077a5cf1471d" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 46803 } } peers { permanent_uuid: "9d2bdbac60244a5d9a356958db1bb1c7" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41705 } } peers { permanent_uuid: "6ae33e71c7d2437abcb65ed59923573d" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 42009 } }
I20260924 16:12:20.436378 463 tablet_bootstrap.cc:492] T 6618a7d814ad4961a5533f57d816e60d P 9d2bdbac60244a5d9a356958db1bb1c7: No bootstrap required, opened a new log
I20260924 16:12:20.436580 463 ts_tablet_manager.cc:1397] T 6618a7d814ad4961a5533f57d816e60d P 9d2bdbac60244a5d9a356958db1bb1c7: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.004s
I20260924 16:12:20.436395 467 tablet_replica.cc:439] T 6618a7d814ad4961a5533f57d816e60d P 445ada5dbd2542b7a204077a5cf1471d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445ada5dbd2542b7a204077a5cf1471d" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 46803 } } peers { permanent_uuid: "9d2bdbac60244a5d9a356958db1bb1c7" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41705 } } peers { permanent_uuid: "6ae33e71c7d2437abcb65ed59923573d" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 42009 } } }
I20260924 16:12:20.436758 467 tablet_replica.cc:442] T 6618a7d814ad4961a5533f57d816e60d P 445ada5dbd2542b7a204077a5cf1471d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:12:20.437513 462 ts_tablet_manager.cc:1428] T 6618a7d814ad4961a5533f57d816e60d P 445ada5dbd2542b7a204077a5cf1471d: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 16:12:20.437577 463 raft_consensus.cc:348] T 6618a7d814ad4961a5533f57d816e60d P 9d2bdbac60244a5d9a356958db1bb1c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445ada5dbd2542b7a204077a5cf1471d" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 46803 } } peers { permanent_uuid: "9d2bdbac60244a5d9a356958db1bb1c7" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41705 } } peers { permanent_uuid: "6ae33e71c7d2437abcb65ed59923573d" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 42009 } }
I20260924 16:12:20.438416 463 raft_consensus.cc:374] T 6618a7d814ad4961a5533f57d816e60d P 9d2bdbac60244a5d9a356958db1bb1c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:20.438526 463 raft_consensus.cc:729] T 6618a7d814ad4961a5533f57d816e60d P 9d2bdbac60244a5d9a356958db1bb1c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d2bdbac60244a5d9a356958db1bb1c7, State: Initialized, Role: FOLLOWER
I20260924 16:12:20.438800 463 consensus_queue.cc:260] T 6618a7d814ad4961a5533f57d816e60d P 9d2bdbac60244a5d9a356958db1bb1c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445ada5dbd2542b7a204077a5cf1471d" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 46803 } } peers { permanent_uuid: "9d2bdbac60244a5d9a356958db1bb1c7" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41705 } } peers { permanent_uuid: "6ae33e71c7d2437abcb65ed59923573d" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 42009 } }
I20260924 16:12:20.439386 463 ts_tablet_manager.cc:1428] T 6618a7d814ad4961a5533f57d816e60d P 9d2bdbac60244a5d9a356958db1bb1c7: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260924 16:12:20.439251 469 tablet_replica.cc:439] T 6618a7d814ad4961a5533f57d816e60d P 9d2bdbac60244a5d9a356958db1bb1c7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445ada5dbd2542b7a204077a5cf1471d" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 46803 } } peers { permanent_uuid: "9d2bdbac60244a5d9a356958db1bb1c7" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41705 } } peers { permanent_uuid: "6ae33e71c7d2437abcb65ed59923573d" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 42009 } } }
I20260924 16:12:20.439642 469 tablet_replica.cc:442] T 6618a7d814ad4961a5533f57d816e60d P 9d2bdbac60244a5d9a356958db1bb1c7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:12:20.540606 468 raft_consensus.cc:482] T 6618a7d814ad4961a5533f57d816e60d P 6ae33e71c7d2437abcb65ed59923573d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:12:20.540827 468 raft_consensus.cc:504] T 6618a7d814ad4961a5533f57d816e60d P 6ae33e71c7d2437abcb65ed59923573d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445ada5dbd2542b7a204077a5cf1471d" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 46803 } } peers { permanent_uuid: "9d2bdbac60244a5d9a356958db1bb1c7" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41705 } } peers { permanent_uuid: "6ae33e71c7d2437abcb65ed59923573d" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 42009 } }
I20260924 16:12:20.541558 468 leader_election.cc:288] T 6618a7d814ad4961a5533f57d816e60d P 6ae33e71c7d2437abcb65ed59923573d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 445ada5dbd2542b7a204077a5cf1471d (127.29.234.1:46803), 9d2bdbac60244a5d9a356958db1bb1c7 (127.29.234.2:41705)
I20260924 16:12:20.542100 32580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6618a7d814ad4961a5533f57d816e60d" candidate_uuid: "6ae33e71c7d2437abcb65ed59923573d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "445ada5dbd2542b7a204077a5cf1471d" is_pre_election: true
I20260924 16:12:20.542142 32717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6618a7d814ad4961a5533f57d816e60d" candidate_uuid: "6ae33e71c7d2437abcb65ed59923573d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d2bdbac60244a5d9a356958db1bb1c7" is_pre_election: true
I20260924 16:12:20.542373 32580 raft_consensus.cc:2454] T 6618a7d814ad4961a5533f57d816e60d P 445ada5dbd2542b7a204077a5cf1471d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ae33e71c7d2437abcb65ed59923573d in term 0.
I20260924 16:12:20.542402 32717 raft_consensus.cc:2454] T 6618a7d814ad4961a5533f57d816e60d P 9d2bdbac60244a5d9a356958db1bb1c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ae33e71c7d2437abcb65ed59923573d in term 0.
I20260924 16:12:20.542794 321 leader_election.cc:302] T 6618a7d814ad4961a5533f57d816e60d P 6ae33e71c7d2437abcb65ed59923573d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 445ada5dbd2542b7a204077a5cf1471d, 6ae33e71c7d2437abcb65ed59923573d; no voters:
I20260924 16:12:20.543100 468 raft_consensus.cc:2781] T 6618a7d814ad4961a5533f57d816e60d P 6ae33e71c7d2437abcb65ed59923573d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:12:20.543190 468 raft_consensus.cc:482] T 6618a7d814ad4961a5533f57d816e60d P 6ae33e71c7d2437abcb65ed59923573d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:12:20.543260 468 raft_consensus.cc:3037] T 6618a7d814ad4961a5533f57d816e60d P 6ae33e71c7d2437abcb65ed59923573d [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:20.545249 468 raft_consensus.cc:504] T 6618a7d814ad4961a5533f57d816e60d P 6ae33e71c7d2437abcb65ed59923573d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445ada5dbd2542b7a204077a5cf1471d" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 46803 } } peers { permanent_uuid: "9d2bdbac60244a5d9a356958db1bb1c7" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41705 } } peers { permanent_uuid: "6ae33e71c7d2437abcb65ed59923573d" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 42009 } }
I20260924 16:12:20.545730 468 leader_election.cc:288] T 6618a7d814ad4961a5533f57d816e60d P 6ae33e71c7d2437abcb65ed59923573d [CANDIDATE]: Term 1 election: Requested vote from peers 445ada5dbd2542b7a204077a5cf1471d (127.29.234.1:46803), 9d2bdbac60244a5d9a356958db1bb1c7 (127.29.234.2:41705)
I20260924 16:12:20.546203 32717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6618a7d814ad4961a5533f57d816e60d" candidate_uuid: "6ae33e71c7d2437abcb65ed59923573d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d2bdbac60244a5d9a356958db1bb1c7"
I20260924 16:12:20.546205 32580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6618a7d814ad4961a5533f57d816e60d" candidate_uuid: "6ae33e71c7d2437abcb65ed59923573d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "445ada5dbd2542b7a204077a5cf1471d"
I20260924 16:12:20.546434 32580 raft_consensus.cc:3037] T 6618a7d814ad4961a5533f57d816e60d P 445ada5dbd2542b7a204077a5cf1471d [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:20.546454 32717 raft_consensus.cc:3037] T 6618a7d814ad4961a5533f57d816e60d P 9d2bdbac60244a5d9a356958db1bb1c7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:20.548413 32580 raft_consensus.cc:2454] T 6618a7d814ad4961a5533f57d816e60d P 445ada5dbd2542b7a204077a5cf1471d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ae33e71c7d2437abcb65ed59923573d in term 1.
I20260924 16:12:20.548476 32717 raft_consensus.cc:2454] T 6618a7d814ad4961a5533f57d816e60d P 9d2bdbac60244a5d9a356958db1bb1c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ae33e71c7d2437abcb65ed59923573d in term 1.
I20260924 16:12:20.548810 321 leader_election.cc:302] T 6618a7d814ad4961a5533f57d816e60d P 6ae33e71c7d2437abcb65ed59923573d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 445ada5dbd2542b7a204077a5cf1471d, 6ae33e71c7d2437abcb65ed59923573d; no voters:
I20260924 16:12:20.549068 468 raft_consensus.cc:2781] T 6618a7d814ad4961a5533f57d816e60d P 6ae33e71c7d2437abcb65ed59923573d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:12:20.549263 468 raft_consensus.cc:686] T 6618a7d814ad4961a5533f57d816e60d P 6ae33e71c7d2437abcb65ed59923573d [term 1 LEADER]: Becoming Leader. State: Replica: 6ae33e71c7d2437abcb65ed59923573d, State: Running, Role: LEADER
I20260924 16:12:20.549621 468 consensus_queue.cc:237] T 6618a7d814ad4961a5533f57d816e60d P 6ae33e71c7d2437abcb65ed59923573d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445ada5dbd2542b7a204077a5cf1471d" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 46803 } } peers { permanent_uuid: "9d2bdbac60244a5d9a356958db1bb1c7" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41705 } } peers { permanent_uuid: "6ae33e71c7d2437abcb65ed59923573d" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 42009 } }
I20260924 16:12:20.551394 471 tablet_replica.cc:439] T 6618a7d814ad4961a5533f57d816e60d P 6ae33e71c7d2437abcb65ed59923573d: TxnStatusTablet state changed. Reason: New leader 6ae33e71c7d2437abcb65ed59923573d. Latest consensus state: current_term: 1 leader_uuid: "6ae33e71c7d2437abcb65ed59923573d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445ada5dbd2542b7a204077a5cf1471d" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 46803 } } peers { permanent_uuid: "9d2bdbac60244a5d9a356958db1bb1c7" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41705 } } peers { permanent_uuid: "6ae33e71c7d2437abcb65ed59923573d" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 42009 } } }
I20260924 16:12:20.551707 471 tablet_replica.cc:442] T 6618a7d814ad4961a5533f57d816e60d P 6ae33e71c7d2437abcb65ed59923573d: This TxnStatusTablet replica's current role is: LEADER
I20260924 16:12:20.552428 483 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 16:12:20.552722 483 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 16:12:20.553059 32413 catalog_manager.cc:5495] T 6618a7d814ad4961a5533f57d816e60d P 6ae33e71c7d2437abcb65ed59923573d reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ae33e71c7d2437abcb65ed59923573d (127.29.234.3). New cstate: current_term: 1 leader_uuid: "6ae33e71c7d2437abcb65ed59923573d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445ada5dbd2542b7a204077a5cf1471d" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 46803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d2bdbac60244a5d9a356958db1bb1c7" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ae33e71c7d2437abcb65ed59923573d" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 42009 } health_report { overall_health: HEALTHY } } }
I20260924 16:12:20.597900 478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:20.590881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.3:42009 (local address 127.0.0.1:37930)
0924 16:12:20.591259 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:20.591280 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:20.591445 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:20.592052 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:20.592060 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:20.592077 (+ 17us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 16:12:20.592340 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:20.592366 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:20.593918 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:20.593924 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:20.595123 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:20.595136 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:20.596197 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:20.596201 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:20.596331 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:20.596398 (+ 67us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 16:12:20.597484 (+ 1086us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 16:12:20.597501 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 16:12:20.597558 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 16:12:20.597670 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186}
I20260924 16:12:20.597982 486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:20.590968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37930 (local address 127.29.234.3:42009)
0924 16:12:20.591465 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:20.591471 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:20.591494 (+ 23us) server_negotiation.cc:407] Connection header received
0924 16:12:20.591696 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:20.591702 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:20.591826 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:20.592016 (+ 190us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:12:20.592524 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:20.593715 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:20.595267 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:20.596041 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:20.596182 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:20.596612 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:12:20.597385 (+ 773us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:12:20.597530 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:20.597581 (+ 51us) server_negotiation.cc:299] Negotiation successful
0924 16:12:20.597671 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":229,"threads_started":1}
I20260924 16:12:20.606061 32580 raft_consensus.cc:1261] T 6618a7d814ad4961a5533f57d816e60d P 445ada5dbd2542b7a204077a5cf1471d [term 1 FOLLOWER]: Refusing update from remote peer 6ae33e71c7d2437abcb65ed59923573d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:12:20.606166 32717 raft_consensus.cc:1261] T 6618a7d814ad4961a5533f57d816e60d P 9d2bdbac60244a5d9a356958db1bb1c7 [term 1 FOLLOWER]: Refusing update from remote peer 6ae33e71c7d2437abcb65ed59923573d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:12:20.607107 471 consensus_queue.cc:1035] T 6618a7d814ad4961a5533f57d816e60d P 6ae33e71c7d2437abcb65ed59923573d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d2bdbac60244a5d9a356958db1bb1c7" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:12:20.607475 468 consensus_queue.cc:1035] T 6618a7d814ad4961a5533f57d816e60d P 6ae33e71c7d2437abcb65ed59923573d [LEADER]: Connected to new peer: Peer: permanent_uuid: "445ada5dbd2542b7a204077a5cf1471d" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 46803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:12:20.620710 469 tablet_replica.cc:439] T 6618a7d814ad4961a5533f57d816e60d P 9d2bdbac60244a5d9a356958db1bb1c7: TxnStatusTablet state changed. Reason: New leader 6ae33e71c7d2437abcb65ed59923573d. Latest consensus state: current_term: 1 leader_uuid: "6ae33e71c7d2437abcb65ed59923573d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445ada5dbd2542b7a204077a5cf1471d" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 46803 } } peers { permanent_uuid: "9d2bdbac60244a5d9a356958db1bb1c7" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41705 } } peers { permanent_uuid: "6ae33e71c7d2437abcb65ed59923573d" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 42009 } } }
I20260924 16:12:20.621037 469 tablet_replica.cc:442] T 6618a7d814ad4961a5533f57d816e60d P 9d2bdbac60244a5d9a356958db1bb1c7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:12:20.624579 467 tablet_replica.cc:439] T 6618a7d814ad4961a5533f57d816e60d P 445ada5dbd2542b7a204077a5cf1471d: TxnStatusTablet state changed. Reason: New leader 6ae33e71c7d2437abcb65ed59923573d. Latest consensus state: current_term: 1 leader_uuid: "6ae33e71c7d2437abcb65ed59923573d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445ada5dbd2542b7a204077a5cf1471d" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 46803 } } peers { permanent_uuid: "9d2bdbac60244a5d9a356958db1bb1c7" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41705 } } peers { permanent_uuid: "6ae33e71c7d2437abcb65ed59923573d" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 42009 } } }
I20260924 16:12:20.624884 467 tablet_replica.cc:442] T 6618a7d814ad4961a5533f57d816e60d P 445ada5dbd2542b7a204077a5cf1471d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:12:20.626029 468 tablet_replica.cc:439] T 6618a7d814ad4961a5533f57d816e60d P 6ae33e71c7d2437abcb65ed59923573d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6ae33e71c7d2437abcb65ed59923573d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445ada5dbd2542b7a204077a5cf1471d" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 46803 } } peers { permanent_uuid: "9d2bdbac60244a5d9a356958db1bb1c7" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41705 } } peers { permanent_uuid: "6ae33e71c7d2437abcb65ed59923573d" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 42009 } } }
I20260924 16:12:20.626458 468 tablet_replica.cc:442] T 6618a7d814ad4961a5533f57d816e60d P 6ae33e71c7d2437abcb65ed59923573d: This TxnStatusTablet replica's current role is: LEADER
I20260924 16:12:20.631155 489 mvcc.cc:205] Tried to move back new op lower bound from 7332930931111550976 to 7332930930893991936. Current Snapshot: MvccSnapshot[applied={T|T < 7332930931111550976}]
I20260924 16:12:20.649857 467 tablet_replica.cc:439] T 6618a7d814ad4961a5533f57d816e60d P 445ada5dbd2542b7a204077a5cf1471d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6ae33e71c7d2437abcb65ed59923573d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445ada5dbd2542b7a204077a5cf1471d" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 46803 } } peers { permanent_uuid: "9d2bdbac60244a5d9a356958db1bb1c7" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41705 } } peers { permanent_uuid: "6ae33e71c7d2437abcb65ed59923573d" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 42009 } } }
I20260924 16:12:20.650182 467 tablet_replica.cc:442] T 6618a7d814ad4961a5533f57d816e60d P 445ada5dbd2542b7a204077a5cf1471d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:12:20.653321 469 tablet_replica.cc:439] T 6618a7d814ad4961a5533f57d816e60d P 9d2bdbac60244a5d9a356958db1bb1c7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6ae33e71c7d2437abcb65ed59923573d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445ada5dbd2542b7a204077a5cf1471d" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 46803 } } peers { permanent_uuid: "9d2bdbac60244a5d9a356958db1bb1c7" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41705 } } peers { permanent_uuid: "6ae33e71c7d2437abcb65ed59923573d" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 42009 } } }
I20260924 16:12:20.653610 469 tablet_replica.cc:442] T 6618a7d814ad4961a5533f57d816e60d P 9d2bdbac60244a5d9a356958db1bb1c7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:12:20.655305 471 tablet_replica.cc:439] T 6618a7d814ad4961a5533f57d816e60d P 6ae33e71c7d2437abcb65ed59923573d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6ae33e71c7d2437abcb65ed59923573d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445ada5dbd2542b7a204077a5cf1471d" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 46803 } } peers { permanent_uuid: "9d2bdbac60244a5d9a356958db1bb1c7" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41705 } } peers { permanent_uuid: "6ae33e71c7d2437abcb65ed59923573d" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 42009 } } }
I20260924 16:12:20.655615 471 tablet_replica.cc:442] T 6618a7d814ad4961a5533f57d816e60d P 6ae33e71c7d2437abcb65ed59923573d: This TxnStatusTablet replica's current role is: LEADER
I20260924 16:12:20.661007 486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:20.640494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37940 (local address 127.29.234.3:42009)
0924 16:12:20.640713 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:20.640722 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:20.651919 (+ 11197us) server_negotiation.cc:407] Connection header received
0924 16:12:20.654708 (+ 2789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:20.654714 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:20.654823 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:20.654958 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:12:20.656771 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:20.657908 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:20.659300 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:20.659723 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:20.659865 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:20.660278 (+ 413us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:12:20.660388 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:12:20.660495 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:20.660632 (+ 137us) server_negotiation.cc:299] Negotiation successful
0924 16:12:20.660712 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 16:12:20.678988 486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:20.672707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.3:42915 (local address 127.29.234.3:42009)
0924 16:12:20.672896 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:20.672901 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:20.673559 (+ 658us) server_negotiation.cc:407] Connection header received
0924 16:12:20.673785 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:20.673790 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:20.673861 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:20.673971 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:12:20.674568 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:20.675403 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:20.676940 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:20.677439 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:20.677582 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:20.678134 (+ 552us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:12:20.678256 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:12:20.678353 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:20.678495 (+ 142us) server_negotiation.cc:299] Negotiation successful
0924 16:12:20.678606 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":52}
I20260924 16:12:20.679047 500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:20.672659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.3:42009 (local address 127.29.234.3:42915)
0924 16:12:20.673464 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:20.673483 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:20.673648 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:20.673999 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:20.674003 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:20.674014 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 16:12:20.674373 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:20.674385 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:20.675549 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:20.675554 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:20.676799 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:20.676812 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:20.677624 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:20.677630 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:20.677876 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:20.677933 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 16:12:20.678351 (+ 418us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 16:12:20.678381 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 16:12:20.678477 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 16:12:20.678652 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"mutex_wait_us":23,"thread_start_us":370,"threads_started":1}
I20260924 16:12:20.689306 32717 raft_consensus.cc:1261] T 6af0d9fd1c4d4dd58ce9d9b2d3637371 P 9d2bdbac60244a5d9a356958db1bb1c7 [term 1 FOLLOWER]: Refusing update from remote peer 6ae33e71c7d2437abcb65ed59923573d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:12:20.689458 32580 raft_consensus.cc:1261] T 6af0d9fd1c4d4dd58ce9d9b2d3637371 P 445ada5dbd2542b7a204077a5cf1471d [term 1 FOLLOWER]: Refusing update from remote peer 6ae33e71c7d2437abcb65ed59923573d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:12:20.689946 471 consensus_queue.cc:1035] T 6af0d9fd1c4d4dd58ce9d9b2d3637371 P 6ae33e71c7d2437abcb65ed59923573d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d2bdbac60244a5d9a356958db1bb1c7" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:12:20.690212 493 consensus_queue.cc:1035] T 6af0d9fd1c4d4dd58ce9d9b2d3637371 P 6ae33e71c7d2437abcb65ed59923573d [LEADER]: Connected to new peer: Peer: permanent_uuid: "445ada5dbd2542b7a204077a5cf1471d" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 46803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:12:20.755811 32415 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58058:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:12:20.756080 32415 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58058:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:12:20.759737 32415 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3838eedbbb89456096bd20aa4891ea73: Sending DeleteTablet for 3 replicas of tablet 6af0d9fd1c4d4dd58ce9d9b2d3637371
I20260924 16:12:20.761049 32697 tablet_service.cc:1514] Processing DeleteTablet for tablet 6af0d9fd1c4d4dd58ce9d9b2d3637371 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:12:20 UTC) from {username='kudu', principal='kudu/127.29.234.62@KRBTEST.COM'} at 127.0.0.1:43082
I20260924 16:12:20.761054 32555 tablet_service.cc:1514] Processing DeleteTablet for tablet 6af0d9fd1c4d4dd58ce9d9b2d3637371 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:12:20 UTC) from {username='kudu', principal='kudu/127.29.234.62@KRBTEST.COM'} at 127.0.0.1:39246
I20260924 16:12:20.761575 369 tablet_service.cc:1514] Processing DeleteTablet for tablet 6af0d9fd1c4d4dd58ce9d9b2d3637371 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:12:20 UTC) from {username='kudu', principal='kudu/127.29.234.62@KRBTEST.COM'} at 127.0.0.1:37924
I20260924 16:12:20.763021 32415 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58058:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:12:20.763899 32415 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:12:20.771458 32554 tablet_service.cc:1467] Processing CreateTablet for tablet ed3aa5943696487a93a462dffbe32613 (DEFAULT_TABLE table=test-table [id=8d9856155b1349b2ad329b1247c73625]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:12:20.771952 368 tablet_service.cc:1467] Processing CreateTablet for tablet ed3aa5943696487a93a462dffbe32613 (DEFAULT_TABLE table=test-table [id=8d9856155b1349b2ad329b1247c73625]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:12:20.772184 32554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed3aa5943696487a93a462dffbe32613. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:20.772486 368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed3aa5943696487a93a462dffbe32613. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:20.775285 32696 tablet_service.cc:1467] Processing CreateTablet for tablet ed3aa5943696487a93a462dffbe32613 (DEFAULT_TABLE table=test-table [id=8d9856155b1349b2ad329b1247c73625]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:12:20.776002 32696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed3aa5943696487a93a462dffbe32613. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:20.782202 461 tablet_bootstrap.cc:492] T ed3aa5943696487a93a462dffbe32613 P 6ae33e71c7d2437abcb65ed59923573d: Bootstrap starting.
I20260924 16:12:20.784031 461 tablet_bootstrap.cc:654] T ed3aa5943696487a93a462dffbe32613 P 6ae33e71c7d2437abcb65ed59923573d: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:20.799611 462 tablet_bootstrap.cc:492] T ed3aa5943696487a93a462dffbe32613 P 445ada5dbd2542b7a204077a5cf1471d: Bootstrap starting.
I20260924 16:12:20.801012 461 tablet_bootstrap.cc:492] T ed3aa5943696487a93a462dffbe32613 P 6ae33e71c7d2437abcb65ed59923573d: No bootstrap required, opened a new log
I20260924 16:12:20.801396 462 tablet_bootstrap.cc:654] T ed3aa5943696487a93a462dffbe32613 P 445ada5dbd2542b7a204077a5cf1471d: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:20.801323 461 ts_tablet_manager.cc:1397] T ed3aa5943696487a93a462dffbe32613 P 6ae33e71c7d2437abcb65ed59923573d: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.003s
I20260924 16:12:20.801738 463 tablet_bootstrap.cc:492] T ed3aa5943696487a93a462dffbe32613 P 9d2bdbac60244a5d9a356958db1bb1c7: Bootstrap starting.
I20260924 16:12:20.802572 461 raft_consensus.cc:348] T ed3aa5943696487a93a462dffbe32613 P 6ae33e71c7d2437abcb65ed59923573d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445ada5dbd2542b7a204077a5cf1471d" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 46803 } } peers { permanent_uuid: "9d2bdbac60244a5d9a356958db1bb1c7" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41705 } } peers { permanent_uuid: "6ae33e71c7d2437abcb65ed59923573d" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 42009 } }
I20260924 16:12:20.802946 461 raft_consensus.cc:374] T ed3aa5943696487a93a462dffbe32613 P 6ae33e71c7d2437abcb65ed59923573d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:20.803073 461 raft_consensus.cc:729] T ed3aa5943696487a93a462dffbe32613 P 6ae33e71c7d2437abcb65ed59923573d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ae33e71c7d2437abcb65ed59923573d, State: Initialized, Role: FOLLOWER
I20260924 16:12:20.803416 461 consensus_queue.cc:260] T ed3aa5943696487a93a462dffbe32613 P 6ae33e71c7d2437abcb65ed59923573d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445ada5dbd2542b7a204077a5cf1471d" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 46803 } } peers { permanent_uuid: "9d2bdbac60244a5d9a356958db1bb1c7" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41705 } } peers { permanent_uuid: "6ae33e71c7d2437abcb65ed59923573d" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 42009 } }
I20260924 16:12:20.803841 463 tablet_bootstrap.cc:654] T ed3aa5943696487a93a462dffbe32613 P 9d2bdbac60244a5d9a356958db1bb1c7: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:20.806871 461 ts_tablet_manager.cc:1428] T ed3aa5943696487a93a462dffbe32613 P 6ae33e71c7d2437abcb65ed59923573d: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260924 16:12:20.809593 507 tablet_replica.cc:330] T 6af0d9fd1c4d4dd58ce9d9b2d3637371 P 6ae33e71c7d2437abcb65ed59923573d: stopping tablet replica
I20260924 16:12:20.810161 507 raft_consensus.cc:2229] T 6af0d9fd1c4d4dd58ce9d9b2d3637371 P 6ae33e71c7d2437abcb65ed59923573d [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:12:20.810868 507 raft_consensus.cc:2258] T 6af0d9fd1c4d4dd58ce9d9b2d3637371 P 6ae33e71c7d2437abcb65ed59923573d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:12:20.812865 507 ts_tablet_manager.cc:1905] T 6af0d9fd1c4d4dd58ce9d9b2d3637371 P 6ae33e71c7d2437abcb65ed59923573d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:12:20.816849 506 tablet_replica.cc:330] T 6af0d9fd1c4d4dd58ce9d9b2d3637371 P 9d2bdbac60244a5d9a356958db1bb1c7: stopping tablet replica
I20260924 16:12:20.817329 506 raft_consensus.cc:2229] T 6af0d9fd1c4d4dd58ce9d9b2d3637371 P 9d2bdbac60244a5d9a356958db1bb1c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:12:20.817791 506 raft_consensus.cc:2258] T 6af0d9fd1c4d4dd58ce9d9b2d3637371 P 9d2bdbac60244a5d9a356958db1bb1c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:12:20.818007 463 tablet_bootstrap.cc:492] T ed3aa5943696487a93a462dffbe32613 P 9d2bdbac60244a5d9a356958db1bb1c7: No bootstrap required, opened a new log
I20260924 16:12:20.818020 462 tablet_bootstrap.cc:492] T ed3aa5943696487a93a462dffbe32613 P 445ada5dbd2542b7a204077a5cf1471d: No bootstrap required, opened a new log
I20260924 16:12:20.818171 462 ts_tablet_manager.cc:1397] T ed3aa5943696487a93a462dffbe32613 P 445ada5dbd2542b7a204077a5cf1471d: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.010s
I20260924 16:12:20.818172 463 ts_tablet_manager.cc:1397] T ed3aa5943696487a93a462dffbe32613 P 9d2bdbac60244a5d9a356958db1bb1c7: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.002s
I20260924 16:12:20.818873 462 raft_consensus.cc:348] T ed3aa5943696487a93a462dffbe32613 P 445ada5dbd2542b7a204077a5cf1471d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445ada5dbd2542b7a204077a5cf1471d" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 46803 } } peers { permanent_uuid: "9d2bdbac60244a5d9a356958db1bb1c7" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41705 } } peers { permanent_uuid: "6ae33e71c7d2437abcb65ed59923573d" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 42009 } }
I20260924 16:12:20.818988 463 raft_consensus.cc:348] T ed3aa5943696487a93a462dffbe32613 P 9d2bdbac60244a5d9a356958db1bb1c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445ada5dbd2542b7a204077a5cf1471d" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 46803 } } peers { permanent_uuid: "9d2bdbac60244a5d9a356958db1bb1c7" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41705 } } peers { permanent_uuid: "6ae33e71c7d2437abcb65ed59923573d" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 42009 } }
I20260924 16:12:20.819129 462 raft_consensus.cc:374] T ed3aa5943696487a93a462dffbe32613 P 445ada5dbd2542b7a204077a5cf1471d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:20.819202 462 raft_consensus.cc:729] T ed3aa5943696487a93a462dffbe32613 P 445ada5dbd2542b7a204077a5cf1471d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 445ada5dbd2542b7a204077a5cf1471d, State: Initialized, Role: FOLLOWER
I20260924 16:12:20.819227 463 raft_consensus.cc:374] T ed3aa5943696487a93a462dffbe32613 P 9d2bdbac60244a5d9a356958db1bb1c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:20.819303 463 raft_consensus.cc:729] T ed3aa5943696487a93a462dffbe32613 P 9d2bdbac60244a5d9a356958db1bb1c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d2bdbac60244a5d9a356958db1bb1c7, State: Initialized, Role: FOLLOWER
I20260924 16:12:20.819473 462 consensus_queue.cc:260] T ed3aa5943696487a93a462dffbe32613 P 445ada5dbd2542b7a204077a5cf1471d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445ada5dbd2542b7a204077a5cf1471d" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 46803 } } peers { permanent_uuid: "9d2bdbac60244a5d9a356958db1bb1c7" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41705 } } peers { permanent_uuid: "6ae33e71c7d2437abcb65ed59923573d" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 42009 } }
I20260924 16:12:20.819568 463 consensus_queue.cc:260] T ed3aa5943696487a93a462dffbe32613 P 9d2bdbac60244a5d9a356958db1bb1c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445ada5dbd2542b7a204077a5cf1471d" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 46803 } } peers { permanent_uuid: "9d2bdbac60244a5d9a356958db1bb1c7" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41705 } } peers { permanent_uuid: "6ae33e71c7d2437abcb65ed59923573d" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 42009 } }
I20260924 16:12:20.820055 462 ts_tablet_manager.cc:1428] T ed3aa5943696487a93a462dffbe32613 P 445ada5dbd2542b7a204077a5cf1471d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 16:12:20.820240 463 ts_tablet_manager.cc:1428] T ed3aa5943696487a93a462dffbe32613 P 9d2bdbac60244a5d9a356958db1bb1c7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 16:12:20.820520 506 ts_tablet_manager.cc:1905] T 6af0d9fd1c4d4dd58ce9d9b2d3637371 P 9d2bdbac60244a5d9a356958db1bb1c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:12:20.820887 505 tablet_replica.cc:330] T 6af0d9fd1c4d4dd58ce9d9b2d3637371 P 445ada5dbd2542b7a204077a5cf1471d: stopping tablet replica
I20260924 16:12:20.821277 505 raft_consensus.cc:2229] T 6af0d9fd1c4d4dd58ce9d9b2d3637371 P 445ada5dbd2542b7a204077a5cf1471d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:12:20.821805 505 raft_consensus.cc:2258] T 6af0d9fd1c4d4dd58ce9d9b2d3637371 P 445ada5dbd2542b7a204077a5cf1471d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:12:20.824792 505 ts_tablet_manager.cc:1905] T 6af0d9fd1c4d4dd58ce9d9b2d3637371 P 445ada5dbd2542b7a204077a5cf1471d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:12:20.828732 506 ts_tablet_manager.cc:1918] T 6af0d9fd1c4d4dd58ce9d9b2d3637371 P 9d2bdbac60244a5d9a356958db1bb1c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 16:12:20.828948 506 log.cc:1196] T 6af0d9fd1c4d4dd58ce9d9b2d3637371 P 9d2bdbac60244a5d9a356958db1bb1c7: Deleting WAL directory at /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/minicluster-data/ts-1/wal/wals/6af0d9fd1c4d4dd58ce9d9b2d3637371
I20260924 16:12:20.829495 506 ts_tablet_manager.cc:1939] T 6af0d9fd1c4d4dd58ce9d9b2d3637371 P 9d2bdbac60244a5d9a356958db1bb1c7: Deleting consensus metadata
I20260924 16:12:20.831346 32401 catalog_manager.cc:4841] TS 9d2bdbac60244a5d9a356958db1bb1c7 (127.29.234.2:41705): tablet 6af0d9fd1c4d4dd58ce9d9b2d3637371 (table test-table [id=8d41c2a8d427470f804dad19c6181bf1]) successfully deleted
I20260924 16:12:20.832583 507 ts_tablet_manager.cc:1918] T 6af0d9fd1c4d4dd58ce9d9b2d3637371 P 6ae33e71c7d2437abcb65ed59923573d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 16:12:20.832702 505 ts_tablet_manager.cc:1918] T 6af0d9fd1c4d4dd58ce9d9b2d3637371 P 445ada5dbd2542b7a204077a5cf1471d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 16:12:20.832863 507 log.cc:1196] T 6af0d9fd1c4d4dd58ce9d9b2d3637371 P 6ae33e71c7d2437abcb65ed59923573d: Deleting WAL directory at /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/minicluster-data/ts-2/wal/wals/6af0d9fd1c4d4dd58ce9d9b2d3637371
I20260924 16:12:20.833036 505 log.cc:1196] T 6af0d9fd1c4d4dd58ce9d9b2d3637371 P 445ada5dbd2542b7a204077a5cf1471d: Deleting WAL directory at /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/minicluster-data/ts-0/wal/wals/6af0d9fd1c4d4dd58ce9d9b2d3637371
I20260924 16:12:20.833431 507 ts_tablet_manager.cc:1939] T 6af0d9fd1c4d4dd58ce9d9b2d3637371 P 6ae33e71c7d2437abcb65ed59923573d: Deleting consensus metadata
I20260924 16:12:20.833554 505 ts_tablet_manager.cc:1939] T 6af0d9fd1c4d4dd58ce9d9b2d3637371 P 445ada5dbd2542b7a204077a5cf1471d: Deleting consensus metadata
I20260924 16:12:20.835202 32400 catalog_manager.cc:4841] TS 445ada5dbd2542b7a204077a5cf1471d (127.29.234.1:46803): tablet 6af0d9fd1c4d4dd58ce9d9b2d3637371 (table test-table [id=8d41c2a8d427470f804dad19c6181bf1]) successfully deleted
I20260924 16:12:20.835457 32400 catalog_manager.cc:4841] TS 6ae33e71c7d2437abcb65ed59923573d (127.29.234.3:42009): tablet 6af0d9fd1c4d4dd58ce9d9b2d3637371 (table test-table [id=8d41c2a8d427470f804dad19c6181bf1]) successfully deleted
I20260924 16:12:21.149570 467 raft_consensus.cc:482] T ed3aa5943696487a93a462dffbe32613 P 445ada5dbd2542b7a204077a5cf1471d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:12:21.149863 467 raft_consensus.cc:504] T ed3aa5943696487a93a462dffbe32613 P 445ada5dbd2542b7a204077a5cf1471d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445ada5dbd2542b7a204077a5cf1471d" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 46803 } } peers { permanent_uuid: "9d2bdbac60244a5d9a356958db1bb1c7" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41705 } } peers { permanent_uuid: "6ae33e71c7d2437abcb65ed59923573d" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 42009 } }
I20260924 16:12:21.151793 467 leader_election.cc:288] T ed3aa5943696487a93a462dffbe32613 P 445ada5dbd2542b7a204077a5cf1471d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d2bdbac60244a5d9a356958db1bb1c7 (127.29.234.2:41705), 6ae33e71c7d2437abcb65ed59923573d (127.29.234.3:42009)
I20260924 16:12:21.159417 511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:21.152102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.2:41705 (local address 127.29.234.1:51563)
0924 16:12:21.152724 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:21.152750 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:21.152902 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:21.153704 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:21.153711 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:21.153746 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:12:21.154063 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:21.154073 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:21.155362 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:21.155370 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:21.156997 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:21.157009 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:21.158593 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:21.158599 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:21.158746 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:21.158788 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 16:12:21.158862 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 16:12:21.158994 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"mutex_wait_us":1,"thread_start_us":518,"threads_started":1}
I20260924 16:12:21.159919 512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:21.152527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.1:51563 (local address 127.29.234.2:41705)
0924 16:12:21.153321 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:21.153327 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:21.153386 (+ 59us) server_negotiation.cc:407] Connection header received
0924 16:12:21.153464 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:21.153470 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:21.153544 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:21.153649 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:12:21.154226 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:21.155197 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:21.157162 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:21.158366 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:21.159319 (+ 953us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:21.159405 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:21.159461 (+ 56us) server_negotiation.cc:299] Negotiation successful
0924 16:12:21.159624 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":674,"thread_start_us":576,"threads_started":1}
I20260924 16:12:21.160547 32717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed3aa5943696487a93a462dffbe32613" candidate_uuid: "445ada5dbd2542b7a204077a5cf1471d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d2bdbac60244a5d9a356958db1bb1c7" is_pre_election: true
I20260924 16:12:21.160802 32717 raft_consensus.cc:2454] T ed3aa5943696487a93a462dffbe32613 P 9d2bdbac60244a5d9a356958db1bb1c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 445ada5dbd2542b7a204077a5cf1471d in term 0.
I20260924 16:12:21.161067 513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:21.152114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.3:42009 (local address 127.29.234.1:47907)
0924 16:12:21.153657 (+ 1543us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:21.153674 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:21.153870 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:21.154263 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:21.154269 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:21.154300 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:12:21.154662 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:21.154674 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:21.156234 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:21.156240 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:21.158404 (+ 2164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:21.158420 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:21.160265 (+ 1845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:21.160272 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:21.160460 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:21.160491 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 16:12:21.160594 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 16:12:21.160709 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1412,"mutex_wait_us":549,"thread_start_us":663,"threads_started":1}
I20260924 16:12:21.161034 486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:21.152424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.1:47907 (local address 127.29.234.3:42009)
0924 16:12:21.152635 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:21.152643 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:21.153751 (+ 1108us) server_negotiation.cc:407] Connection header received
0924 16:12:21.154033 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:21.154040 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:21.154115 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:21.154203 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:12:21.154822 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:21.156032 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:21.158587 (+ 2555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:21.159948 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:21.160114 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:21.160189 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:21.160623 (+ 434us) server_negotiation.cc:299] Negotiation successful
0924 16:12:21.160745 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 16:12:21.161557 32506 leader_election.cc:302] T ed3aa5943696487a93a462dffbe32613 P 445ada5dbd2542b7a204077a5cf1471d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 445ada5dbd2542b7a204077a5cf1471d, 9d2bdbac60244a5d9a356958db1bb1c7; no voters:
I20260924 16:12:21.161965 467 raft_consensus.cc:2781] T ed3aa5943696487a93a462dffbe32613 P 445ada5dbd2542b7a204077a5cf1471d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:12:21.162102 467 raft_consensus.cc:482] T ed3aa5943696487a93a462dffbe32613 P 445ada5dbd2542b7a204077a5cf1471d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:12:21.162206 467 raft_consensus.cc:3037] T ed3aa5943696487a93a462dffbe32613 P 445ada5dbd2542b7a204077a5cf1471d [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:21.162130 396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed3aa5943696487a93a462dffbe32613" candidate_uuid: "445ada5dbd2542b7a204077a5cf1471d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ae33e71c7d2437abcb65ed59923573d" is_pre_election: true
I20260924 16:12:21.162600 396 raft_consensus.cc:2454] T ed3aa5943696487a93a462dffbe32613 P 6ae33e71c7d2437abcb65ed59923573d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 445ada5dbd2542b7a204077a5cf1471d in term 0.
I20260924 16:12:21.164671 467 raft_consensus.cc:504] T ed3aa5943696487a93a462dffbe32613 P 445ada5dbd2542b7a204077a5cf1471d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445ada5dbd2542b7a204077a5cf1471d" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 46803 } } peers { permanent_uuid: "9d2bdbac60244a5d9a356958db1bb1c7" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41705 } } peers { permanent_uuid: "6ae33e71c7d2437abcb65ed59923573d" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 42009 } }
I20260924 16:12:21.165268 467 leader_election.cc:288] T ed3aa5943696487a93a462dffbe32613 P 445ada5dbd2542b7a204077a5cf1471d [CANDIDATE]: Term 1 election: Requested vote from peers 9d2bdbac60244a5d9a356958db1bb1c7 (127.29.234.2:41705), 6ae33e71c7d2437abcb65ed59923573d (127.29.234.3:42009)
I20260924 16:12:21.165745 32717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed3aa5943696487a93a462dffbe32613" candidate_uuid: "445ada5dbd2542b7a204077a5cf1471d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d2bdbac60244a5d9a356958db1bb1c7"
I20260924 16:12:21.165794 396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed3aa5943696487a93a462dffbe32613" candidate_uuid: "445ada5dbd2542b7a204077a5cf1471d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ae33e71c7d2437abcb65ed59923573d"
I20260924 16:12:21.165939 32717 raft_consensus.cc:3037] T ed3aa5943696487a93a462dffbe32613 P 9d2bdbac60244a5d9a356958db1bb1c7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:21.165988 396 raft_consensus.cc:3037] T ed3aa5943696487a93a462dffbe32613 P 6ae33e71c7d2437abcb65ed59923573d [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:21.167842 32717 raft_consensus.cc:2454] T ed3aa5943696487a93a462dffbe32613 P 9d2bdbac60244a5d9a356958db1bb1c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 445ada5dbd2542b7a204077a5cf1471d in term 1.
I20260924 16:12:21.168195 32506 leader_election.cc:302] T ed3aa5943696487a93a462dffbe32613 P 445ada5dbd2542b7a204077a5cf1471d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 445ada5dbd2542b7a204077a5cf1471d, 9d2bdbac60244a5d9a356958db1bb1c7; no voters:
I20260924 16:12:21.168502 467 raft_consensus.cc:2781] T ed3aa5943696487a93a462dffbe32613 P 445ada5dbd2542b7a204077a5cf1471d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:12:21.169041 467 raft_consensus.cc:686] T ed3aa5943696487a93a462dffbe32613 P 445ada5dbd2542b7a204077a5cf1471d [term 1 LEADER]: Becoming Leader. State: Replica: 445ada5dbd2542b7a204077a5cf1471d, State: Running, Role: LEADER
I20260924 16:12:21.169540 396 raft_consensus.cc:2454] T ed3aa5943696487a93a462dffbe32613 P 6ae33e71c7d2437abcb65ed59923573d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 445ada5dbd2542b7a204077a5cf1471d in term 1.
I20260924 16:12:21.169534 467 consensus_queue.cc:237] T ed3aa5943696487a93a462dffbe32613 P 445ada5dbd2542b7a204077a5cf1471d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445ada5dbd2542b7a204077a5cf1471d" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 46803 } } peers { permanent_uuid: "9d2bdbac60244a5d9a356958db1bb1c7" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41705 } } peers { permanent_uuid: "6ae33e71c7d2437abcb65ed59923573d" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 42009 } }
I20260924 16:12:21.173178 32415 catalog_manager.cc:5495] T ed3aa5943696487a93a462dffbe32613 P 445ada5dbd2542b7a204077a5cf1471d reported cstate change: term changed from 0 to 1, leader changed from <none> to 445ada5dbd2542b7a204077a5cf1471d (127.29.234.1). New cstate: current_term: 1 leader_uuid: "445ada5dbd2542b7a204077a5cf1471d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445ada5dbd2542b7a204077a5cf1471d" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 46803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d2bdbac60244a5d9a356958db1bb1c7" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ae33e71c7d2437abcb65ed59923573d" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 42009 } health_report { overall_health: UNKNOWN } } }
I20260924 16:12:21.291229 525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:21.271461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39256 (local address 127.29.234.1:46803)
0924 16:12:21.284290 (+ 12829us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:21.284296 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:21.284942 (+ 646us) server_negotiation.cc:407] Connection header received
0924 16:12:21.285177 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:21.285181 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:21.285261 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:21.285493 (+ 232us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:12:21.285964 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:21.286731 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:21.288739 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:21.289154 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:21.289247 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:21.289828 (+ 581us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:12:21.290474 (+ 646us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:12:21.290578 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:21.290768 (+ 190us) server_negotiation.cc:299] Negotiation successful
0924 16:12:21.290839 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12577,"thread_start_us":12372,"threads_started":1}
I20260924 16:12:21.311182 486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:21.303615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.1:56465 (local address 127.29.234.3:42009)
0924 16:12:21.303864 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:21.303879 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:21.306032 (+ 2153us) server_negotiation.cc:407] Connection header received
0924 16:12:21.306278 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:21.306285 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:21.306378 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:21.306479 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:12:21.306950 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:21.307763 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:21.309149 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:21.309828 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:21.309988 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:21.310404 (+ 416us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:12:21.310508 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:12:21.310616 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:21.310744 (+ 128us) server_negotiation.cc:299] Negotiation successful
0924 16:12:21.310827 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 16:12:21.311269 529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:21.303567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.3:42009 (local address 127.29.234.1:56465)
0924 16:12:21.305926 (+ 2359us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:21.305942 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:21.306115 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:21.306494 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:21.306499 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:21.306510 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 16:12:21.306811 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:21.306819 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:21.307948 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:21.307953 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:21.308999 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:21.309011 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:21.310024 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:21.310031 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:21.310168 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:21.310222 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 16:12:21.310611 (+ 389us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 16:12:21.310625 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 16:12:21.310724 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 16:12:21.310856 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2210,"thread_start_us":2021,"threads_started":1}
I20260924 16:12:21.323428 32716 raft_consensus.cc:1261] T ed3aa5943696487a93a462dffbe32613 P 9d2bdbac60244a5d9a356958db1bb1c7 [term 1 FOLLOWER]: Refusing update from remote peer 445ada5dbd2542b7a204077a5cf1471d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:12:21.323794 396 raft_consensus.cc:1261] T ed3aa5943696487a93a462dffbe32613 P 6ae33e71c7d2437abcb65ed59923573d [term 1 FOLLOWER]: Refusing update from remote peer 445ada5dbd2542b7a204077a5cf1471d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:12:21.324198 467 consensus_queue.cc:1035] T ed3aa5943696487a93a462dffbe32613 P 445ada5dbd2542b7a204077a5cf1471d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d2bdbac60244a5d9a356958db1bb1c7" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:12:21.324586 514 consensus_queue.cc:1035] T ed3aa5943696487a93a462dffbe32613 P 445ada5dbd2542b7a204077a5cf1471d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ae33e71c7d2437abcb65ed59923573d" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 42009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 16:12:21.378868 525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:21.373423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.3:57275 (local address 127.29.234.1:46803)
0924 16:12:21.373667 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:21.373675 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:21.374083 (+ 408us) server_negotiation.cc:407] Connection header received
0924 16:12:21.374285 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:21.374291 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:21.374374 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:21.374469 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:12:21.374971 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:21.375664 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:21.377106 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:21.377588 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:21.377710 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:21.378164 (+ 454us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:12:21.378263 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:12:21.378364 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:21.378476 (+ 112us) server_negotiation.cc:299] Negotiation successful
0924 16:12:21.378552 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 16:12:21.379007 536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:21.373277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:46803 (local address 127.29.234.3:57275)
0924 16:12:21.373974 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:21.373996 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:21.374147 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:21.374485 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:21.374489 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:21.374505 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 16:12:21.374809 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:21.374825 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:21.375793 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:21.375799 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:21.376962 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:21.376973 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:21.377780 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:21.377790 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:21.377931 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:21.377967 (+ 36us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 16:12:21.378359 (+ 392us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 16:12:21.378372 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 16:12:21.378447 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 16:12:21.378585 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":434,"threads_started":1}
W20260924 16:12:21.380196 518 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 16:12:23.009552 32415 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58058:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:12:23.009786 32415 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58058:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:12:23.013947 32415 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3838eedbbb89456096bd20aa4891ea73: Sending DeleteTablet for 3 replicas of tablet ed3aa5943696487a93a462dffbe32613
I20260924 16:12:23.014760 32555 tablet_service.cc:1514] Processing DeleteTablet for tablet ed3aa5943696487a93a462dffbe32613 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:12:23 UTC) from {username='kudu', principal='kudu/127.29.234.62@KRBTEST.COM'} at 127.0.0.1:39246
I20260924 16:12:23.014875 369 tablet_service.cc:1514] Processing DeleteTablet for tablet ed3aa5943696487a93a462dffbe32613 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:12:23 UTC) from {username='kudu', principal='kudu/127.29.234.62@KRBTEST.COM'} at 127.0.0.1:37924
I20260924 16:12:23.014878 32697 tablet_service.cc:1514] Processing DeleteTablet for tablet ed3aa5943696487a93a462dffbe32613 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:12:23 UTC) from {username='kudu', principal='kudu/127.29.234.62@KRBTEST.COM'} at 127.0.0.1:43082
I20260924 16:12:23.027357 561 tablet_replica.cc:330] T ed3aa5943696487a93a462dffbe32613 P 6ae33e71c7d2437abcb65ed59923573d: stopping tablet replica
I20260924 16:12:23.028010 560 tablet_replica.cc:330] T ed3aa5943696487a93a462dffbe32613 P 445ada5dbd2542b7a204077a5cf1471d: stopping tablet replica
I20260924 16:12:23.028330 562 tablet_replica.cc:330] T ed3aa5943696487a93a462dffbe32613 P 9d2bdbac60244a5d9a356958db1bb1c7: stopping tablet replica
I20260924 16:12:23.028302 560 raft_consensus.cc:2229] T ed3aa5943696487a93a462dffbe32613 P 445ada5dbd2542b7a204077a5cf1471d [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:12:23.028370 561 raft_consensus.cc:2229] T ed3aa5943696487a93a462dffbe32613 P 6ae33e71c7d2437abcb65ed59923573d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:12:23.028568 562 raft_consensus.cc:2229] T ed3aa5943696487a93a462dffbe32613 P 9d2bdbac60244a5d9a356958db1bb1c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:12:23.028728 561 raft_consensus.cc:2258] T ed3aa5943696487a93a462dffbe32613 P 6ae33e71c7d2437abcb65ed59923573d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:12:23.028864 562 raft_consensus.cc:2258] T ed3aa5943696487a93a462dffbe32613 P 9d2bdbac60244a5d9a356958db1bb1c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:12:23.029477 560 raft_consensus.cc:2258] T ed3aa5943696487a93a462dffbe32613 P 445ada5dbd2542b7a204077a5cf1471d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:12:23.029948 561 ts_tablet_manager.cc:1905] T ed3aa5943696487a93a462dffbe32613 P 6ae33e71c7d2437abcb65ed59923573d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:12:23.029994 562 ts_tablet_manager.cc:1905] T ed3aa5943696487a93a462dffbe32613 P 9d2bdbac60244a5d9a356958db1bb1c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:12:23.034065 560 ts_tablet_manager.cc:1905] T ed3aa5943696487a93a462dffbe32613 P 445ada5dbd2542b7a204077a5cf1471d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:12:23.034838 561 ts_tablet_manager.cc:1918] T ed3aa5943696487a93a462dffbe32613 P 6ae33e71c7d2437abcb65ed59923573d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 16:12:23.035030 561 log.cc:1196] T ed3aa5943696487a93a462dffbe32613 P 6ae33e71c7d2437abcb65ed59923573d: Deleting WAL directory at /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/minicluster-data/ts-2/wal/wals/ed3aa5943696487a93a462dffbe32613
I20260924 16:12:23.035571 561 ts_tablet_manager.cc:1939] T ed3aa5943696487a93a462dffbe32613 P 6ae33e71c7d2437abcb65ed59923573d: Deleting consensus metadata
I20260924 16:12:23.036885 32400 catalog_manager.cc:4841] TS 6ae33e71c7d2437abcb65ed59923573d (127.29.234.3:42009): tablet ed3aa5943696487a93a462dffbe32613 (table test-table [id=8d9856155b1349b2ad329b1247c73625]) successfully deleted
I20260924 16:12:23.037643 562 ts_tablet_manager.cc:1918] T ed3aa5943696487a93a462dffbe32613 P 9d2bdbac60244a5d9a356958db1bb1c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 16:12:23.037860 562 log.cc:1196] T ed3aa5943696487a93a462dffbe32613 P 9d2bdbac60244a5d9a356958db1bb1c7: Deleting WAL directory at /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/minicluster-data/ts-1/wal/wals/ed3aa5943696487a93a462dffbe32613
I20260924 16:12:23.038337 562 ts_tablet_manager.cc:1939] T ed3aa5943696487a93a462dffbe32613 P 9d2bdbac60244a5d9a356958db1bb1c7: Deleting consensus metadata
I20260924 16:12:23.039575 32401 catalog_manager.cc:4841] TS 9d2bdbac60244a5d9a356958db1bb1c7 (127.29.234.2:41705): tablet ed3aa5943696487a93a462dffbe32613 (table test-table [id=8d9856155b1349b2ad329b1247c73625]) successfully deleted
I20260924 16:12:23.040120 560 ts_tablet_manager.cc:1918] T ed3aa5943696487a93a462dffbe32613 P 445ada5dbd2542b7a204077a5cf1471d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 16:12:23.040272 560 log.cc:1196] T ed3aa5943696487a93a462dffbe32613 P 445ada5dbd2542b7a204077a5cf1471d: Deleting WAL directory at /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/minicluster-data/ts-0/wal/wals/ed3aa5943696487a93a462dffbe32613
I20260924 16:12:23.040720 560 ts_tablet_manager.cc:1939] T ed3aa5943696487a93a462dffbe32613 P 445ada5dbd2542b7a204077a5cf1471d: Deleting consensus metadata
I20260924 16:12:23.041913 32400 catalog_manager.cc:4841] TS 445ada5dbd2542b7a204077a5cf1471d (127.29.234.1:46803): tablet ed3aa5943696487a93a462dffbe32613 (table test-table [id=8d9856155b1349b2ad329b1247c73625]) successfully deleted
Sep 24 16:12:23 dist-test-slave-v5j9 krb5kdc[32370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266343, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 16:12:23 dist-test-slave-v5j9 krb5kdc[32370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266343, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:12:23.074496 570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:23.060127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34074 (local address 127.29.234.62:39565)
0924 16:12:23.060723 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:23.060727 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:23.060749 (+ 22us) server_negotiation.cc:407] Connection header received
0924 16:12:23.060888 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:23.060892 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:23.060952 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:23.061044 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:23.062383 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:23.063205 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:23.064451 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:23.064948 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:23.065053 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:23.068880 (+ 3827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:23.068885 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:23.068888 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:23.068920 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:23.071742 (+ 2822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:23.072349 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:23.072353 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:23.072355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:23.072511 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:23.072942 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:23.072947 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:23.072950 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:23.073313 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:23.073581 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:23.073925 (+ 344us) server_negotiation.cc:299] Negotiation successful
0924 16:12:23.074158 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":331,"threads_started":1}
I20260924 16:12:23.078120 32415 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34074:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:12:23.078807 32415 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:12:23.085557 32555 tablet_service.cc:1467] Processing CreateTablet for tablet 93c25f9445384aa2b2faa810f304bf94 (DEFAULT_TABLE table=test-table [id=c78e2e40dc87415191fd102b6a816dd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:12:23.085632 32697 tablet_service.cc:1467] Processing CreateTablet for tablet 93c25f9445384aa2b2faa810f304bf94 (DEFAULT_TABLE table=test-table [id=c78e2e40dc87415191fd102b6a816dd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:12:23.086035 32697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93c25f9445384aa2b2faa810f304bf94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:23.086045 32555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93c25f9445384aa2b2faa810f304bf94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:23.085987 369 tablet_service.cc:1467] Processing CreateTablet for tablet 93c25f9445384aa2b2faa810f304bf94 (DEFAULT_TABLE table=test-table [id=c78e2e40dc87415191fd102b6a816dd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:12:23.086397 369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93c25f9445384aa2b2faa810f304bf94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:23.090900 571 tablet_bootstrap.cc:492] T 93c25f9445384aa2b2faa810f304bf94 P 9d2bdbac60244a5d9a356958db1bb1c7: Bootstrap starting.
I20260924 16:12:23.092705 571 tablet_bootstrap.cc:654] T 93c25f9445384aa2b2faa810f304bf94 P 9d2bdbac60244a5d9a356958db1bb1c7: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:23.094972 571 tablet_bootstrap.cc:492] T 93c25f9445384aa2b2faa810f304bf94 P 9d2bdbac60244a5d9a356958db1bb1c7: No bootstrap required, opened a new log
I20260924 16:12:23.095111 571 ts_tablet_manager.cc:1397] T 93c25f9445384aa2b2faa810f304bf94 P 9d2bdbac60244a5d9a356958db1bb1c7: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260924 16:12:23.095144 573 tablet_bootstrap.cc:492] T 93c25f9445384aa2b2faa810f304bf94 P 6ae33e71c7d2437abcb65ed59923573d: Bootstrap starting.
I20260924 16:12:23.095808 571 raft_consensus.cc:348] T 93c25f9445384aa2b2faa810f304bf94 P 9d2bdbac60244a5d9a356958db1bb1c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445ada5dbd2542b7a204077a5cf1471d" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 46803 } } peers { permanent_uuid: "9d2bdbac60244a5d9a356958db1bb1c7" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41705 } } peers { permanent_uuid: "6ae33e71c7d2437abcb65ed59923573d" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 42009 } }
I20260924 16:12:23.095981 571 raft_consensus.cc:374] T 93c25f9445384aa2b2faa810f304bf94 P 9d2bdbac60244a5d9a356958db1bb1c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:23.096042 571 raft_consensus.cc:729] T 93c25f9445384aa2b2faa810f304bf94 P 9d2bdbac60244a5d9a356958db1bb1c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d2bdbac60244a5d9a356958db1bb1c7, State: Initialized, Role: FOLLOWER
I20260924 16:12:23.096212 571 consensus_queue.cc:260] T 93c25f9445384aa2b2faa810f304bf94 P 9d2bdbac60244a5d9a356958db1bb1c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445ada5dbd2542b7a204077a5cf1471d" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 46803 } } peers { permanent_uuid: "9d2bdbac60244a5d9a356958db1bb1c7" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41705 } } peers { permanent_uuid: "6ae33e71c7d2437abcb65ed59923573d" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 42009 } }
I20260924 16:12:23.096567 573 tablet_bootstrap.cc:654] T 93c25f9445384aa2b2faa810f304bf94 P 6ae33e71c7d2437abcb65ed59923573d: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:23.096958 571 ts_tablet_manager.cc:1428] T 93c25f9445384aa2b2faa810f304bf94 P 9d2bdbac60244a5d9a356958db1bb1c7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 16:12:23.097136 572 tablet_bootstrap.cc:492] T 93c25f9445384aa2b2faa810f304bf94 P 445ada5dbd2542b7a204077a5cf1471d: Bootstrap starting.
I20260924 16:12:23.099747 572 tablet_bootstrap.cc:654] T 93c25f9445384aa2b2faa810f304bf94 P 445ada5dbd2542b7a204077a5cf1471d: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:23.101959 572 tablet_bootstrap.cc:492] T 93c25f9445384aa2b2faa810f304bf94 P 445ada5dbd2542b7a204077a5cf1471d: No bootstrap required, opened a new log
I20260924 16:12:23.102051 573 tablet_bootstrap.cc:492] T 93c25f9445384aa2b2faa810f304bf94 P 6ae33e71c7d2437abcb65ed59923573d: No bootstrap required, opened a new log
I20260924 16:12:23.102110 572 ts_tablet_manager.cc:1397] T 93c25f9445384aa2b2faa810f304bf94 P 445ada5dbd2542b7a204077a5cf1471d: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260924 16:12:23.102214 573 ts_tablet_manager.cc:1397] T 93c25f9445384aa2b2faa810f304bf94 P 6ae33e71c7d2437abcb65ed59923573d: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260924 16:12:23.102802 572 raft_consensus.cc:348] T 93c25f9445384aa2b2faa810f304bf94 P 445ada5dbd2542b7a204077a5cf1471d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445ada5dbd2542b7a204077a5cf1471d" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 46803 } } peers { permanent_uuid: "9d2bdbac60244a5d9a356958db1bb1c7" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41705 } } peers { permanent_uuid: "6ae33e71c7d2437abcb65ed59923573d" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 42009 } }
I20260924 16:12:23.102825 573 raft_consensus.cc:348] T 93c25f9445384aa2b2faa810f304bf94 P 6ae33e71c7d2437abcb65ed59923573d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445ada5dbd2542b7a204077a5cf1471d" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 46803 } } peers { permanent_uuid: "9d2bdbac60244a5d9a356958db1bb1c7" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41705 } } peers { permanent_uuid: "6ae33e71c7d2437abcb65ed59923573d" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 42009 } }
I20260924 16:12:23.102993 572 raft_consensus.cc:374] T 93c25f9445384aa2b2faa810f304bf94 P 445ada5dbd2542b7a204077a5cf1471d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:23.103072 573 raft_consensus.cc:374] T 93c25f9445384aa2b2faa810f304bf94 P 6ae33e71c7d2437abcb65ed59923573d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:23.103103 572 raft_consensus.cc:729] T 93c25f9445384aa2b2faa810f304bf94 P 445ada5dbd2542b7a204077a5cf1471d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 445ada5dbd2542b7a204077a5cf1471d, State: Initialized, Role: FOLLOWER
I20260924 16:12:23.103165 573 raft_consensus.cc:729] T 93c25f9445384aa2b2faa810f304bf94 P 6ae33e71c7d2437abcb65ed59923573d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ae33e71c7d2437abcb65ed59923573d, State: Initialized, Role: FOLLOWER
I20260924 16:12:23.103410 573 consensus_queue.cc:260] T 93c25f9445384aa2b2faa810f304bf94 P 6ae33e71c7d2437abcb65ed59923573d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445ada5dbd2542b7a204077a5cf1471d" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 46803 } } peers { permanent_uuid: "9d2bdbac60244a5d9a356958db1bb1c7" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41705 } } peers { permanent_uuid: "6ae33e71c7d2437abcb65ed59923573d" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 42009 } }
I20260924 16:12:23.103400 572 consensus_queue.cc:260] T 93c25f9445384aa2b2faa810f304bf94 P 445ada5dbd2542b7a204077a5cf1471d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445ada5dbd2542b7a204077a5cf1471d" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 46803 } } peers { permanent_uuid: "9d2bdbac60244a5d9a356958db1bb1c7" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41705 } } peers { permanent_uuid: "6ae33e71c7d2437abcb65ed59923573d" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 42009 } }
I20260924 16:12:23.103989 572 ts_tablet_manager.cc:1428] T 93c25f9445384aa2b2faa810f304bf94 P 445ada5dbd2542b7a204077a5cf1471d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260924 16:12:23.103995 573 ts_tablet_manager.cc:1428] T 93c25f9445384aa2b2faa810f304bf94 P 6ae33e71c7d2437abcb65ed59923573d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260924 16:12:23.261314 517 raft_consensus.cc:482] T 93c25f9445384aa2b2faa810f304bf94 P 6ae33e71c7d2437abcb65ed59923573d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:12:23.261698 517 raft_consensus.cc:504] T 93c25f9445384aa2b2faa810f304bf94 P 6ae33e71c7d2437abcb65ed59923573d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445ada5dbd2542b7a204077a5cf1471d" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 46803 } } peers { permanent_uuid: "9d2bdbac60244a5d9a356958db1bb1c7" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41705 } } peers { permanent_uuid: "6ae33e71c7d2437abcb65ed59923573d" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 42009 } }
I20260924 16:12:23.262476 517 leader_election.cc:288] T 93c25f9445384aa2b2faa810f304bf94 P 6ae33e71c7d2437abcb65ed59923573d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 445ada5dbd2542b7a204077a5cf1471d (127.29.234.1:46803), 9d2bdbac60244a5d9a356958db1bb1c7 (127.29.234.2:41705)
I20260924 16:12:23.263033 32580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93c25f9445384aa2b2faa810f304bf94" candidate_uuid: "6ae33e71c7d2437abcb65ed59923573d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "445ada5dbd2542b7a204077a5cf1471d" is_pre_election: true
I20260924 16:12:23.263049 32717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93c25f9445384aa2b2faa810f304bf94" candidate_uuid: "6ae33e71c7d2437abcb65ed59923573d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d2bdbac60244a5d9a356958db1bb1c7" is_pre_election: true
I20260924 16:12:23.263286 32580 raft_consensus.cc:2454] T 93c25f9445384aa2b2faa810f304bf94 P 445ada5dbd2542b7a204077a5cf1471d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ae33e71c7d2437abcb65ed59923573d in term 0.
I20260924 16:12:23.263446 32717 raft_consensus.cc:2454] T 93c25f9445384aa2b2faa810f304bf94 P 9d2bdbac60244a5d9a356958db1bb1c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ae33e71c7d2437abcb65ed59923573d in term 0.
I20260924 16:12:23.263808 321 leader_election.cc:302] T 93c25f9445384aa2b2faa810f304bf94 P 6ae33e71c7d2437abcb65ed59923573d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 445ada5dbd2542b7a204077a5cf1471d, 6ae33e71c7d2437abcb65ed59923573d; no voters:
I20260924 16:12:23.264217 517 raft_consensus.cc:2781] T 93c25f9445384aa2b2faa810f304bf94 P 6ae33e71c7d2437abcb65ed59923573d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:12:23.264339 517 raft_consensus.cc:482] T 93c25f9445384aa2b2faa810f304bf94 P 6ae33e71c7d2437abcb65ed59923573d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:12:23.264411 517 raft_consensus.cc:3037] T 93c25f9445384aa2b2faa810f304bf94 P 6ae33e71c7d2437abcb65ed59923573d [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:23.266546 517 raft_consensus.cc:504] T 93c25f9445384aa2b2faa810f304bf94 P 6ae33e71c7d2437abcb65ed59923573d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445ada5dbd2542b7a204077a5cf1471d" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 46803 } } peers { permanent_uuid: "9d2bdbac60244a5d9a356958db1bb1c7" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41705 } } peers { permanent_uuid: "6ae33e71c7d2437abcb65ed59923573d" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 42009 } }
I20260924 16:12:23.267086 517 leader_election.cc:288] T 93c25f9445384aa2b2faa810f304bf94 P 6ae33e71c7d2437abcb65ed59923573d [CANDIDATE]: Term 1 election: Requested vote from peers 445ada5dbd2542b7a204077a5cf1471d (127.29.234.1:46803), 9d2bdbac60244a5d9a356958db1bb1c7 (127.29.234.2:41705)
I20260924 16:12:23.267498 32580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93c25f9445384aa2b2faa810f304bf94" candidate_uuid: "6ae33e71c7d2437abcb65ed59923573d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "445ada5dbd2542b7a204077a5cf1471d"
I20260924 16:12:23.267558 32717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93c25f9445384aa2b2faa810f304bf94" candidate_uuid: "6ae33e71c7d2437abcb65ed59923573d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d2bdbac60244a5d9a356958db1bb1c7"
I20260924 16:12:23.267733 32580 raft_consensus.cc:3037] T 93c25f9445384aa2b2faa810f304bf94 P 445ada5dbd2542b7a204077a5cf1471d [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:23.267807 32717 raft_consensus.cc:3037] T 93c25f9445384aa2b2faa810f304bf94 P 9d2bdbac60244a5d9a356958db1bb1c7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:23.270133 32580 raft_consensus.cc:2454] T 93c25f9445384aa2b2faa810f304bf94 P 445ada5dbd2542b7a204077a5cf1471d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ae33e71c7d2437abcb65ed59923573d in term 1.
I20260924 16:12:23.270157 32717 raft_consensus.cc:2454] T 93c25f9445384aa2b2faa810f304bf94 P 9d2bdbac60244a5d9a356958db1bb1c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ae33e71c7d2437abcb65ed59923573d in term 1.
I20260924 16:12:23.270563 322 leader_election.cc:302] T 93c25f9445384aa2b2faa810f304bf94 P 6ae33e71c7d2437abcb65ed59923573d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ae33e71c7d2437abcb65ed59923573d, 9d2bdbac60244a5d9a356958db1bb1c7; no voters:
I20260924 16:12:23.270871 517 raft_consensus.cc:2781] T 93c25f9445384aa2b2faa810f304bf94 P 6ae33e71c7d2437abcb65ed59923573d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:12:23.271030 517 raft_consensus.cc:686] T 93c25f9445384aa2b2faa810f304bf94 P 6ae33e71c7d2437abcb65ed59923573d [term 1 LEADER]: Becoming Leader. State: Replica: 6ae33e71c7d2437abcb65ed59923573d, State: Running, Role: LEADER
I20260924 16:12:23.271284 517 consensus_queue.cc:237] T 93c25f9445384aa2b2faa810f304bf94 P 6ae33e71c7d2437abcb65ed59923573d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445ada5dbd2542b7a204077a5cf1471d" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 46803 } } peers { permanent_uuid: "9d2bdbac60244a5d9a356958db1bb1c7" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41705 } } peers { permanent_uuid: "6ae33e71c7d2437abcb65ed59923573d" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 42009 } }
I20260924 16:12:23.274246 32413 catalog_manager.cc:5495] T 93c25f9445384aa2b2faa810f304bf94 P 6ae33e71c7d2437abcb65ed59923573d reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ae33e71c7d2437abcb65ed59923573d (127.29.234.3). New cstate: current_term: 1 leader_uuid: "6ae33e71c7d2437abcb65ed59923573d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445ada5dbd2542b7a204077a5cf1471d" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 46803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d2bdbac60244a5d9a356958db1bb1c7" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ae33e71c7d2437abcb65ed59923573d" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 42009 } health_report { overall_health: HEALTHY } } }
I20260924 16:12:23.300678 581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:23.295851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42940 (local address 127.29.234.3:42009)
0924 16:12:23.296372 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:23.296376 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:23.296395 (+ 19us) server_negotiation.cc:407] Connection header received
0924 16:12:23.296445 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:23.296449 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:23.296511 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:23.296591 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:12:23.296907 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:23.297767 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:23.299084 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:23.299504 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:23.299601 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:23.299970 (+ 369us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:12:23.300065 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:12:23.300129 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:23.300266 (+ 137us) server_negotiation.cc:299] Negotiation successful
0924 16:12:23.300325 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":277,"threads_started":1}
I20260924 16:12:23.310267 32717 raft_consensus.cc:1261] T 93c25f9445384aa2b2faa810f304bf94 P 9d2bdbac60244a5d9a356958db1bb1c7 [term 1 FOLLOWER]: Refusing update from remote peer 6ae33e71c7d2437abcb65ed59923573d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:12:23.310236 32580 raft_consensus.cc:1261] T 93c25f9445384aa2b2faa810f304bf94 P 445ada5dbd2542b7a204077a5cf1471d [term 1 FOLLOWER]: Refusing update from remote peer 6ae33e71c7d2437abcb65ed59923573d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:12:23.310807 579 consensus_queue.cc:1035] T 93c25f9445384aa2b2faa810f304bf94 P 6ae33e71c7d2437abcb65ed59923573d [LEADER]: Connected to new peer: Peer: permanent_uuid: "445ada5dbd2542b7a204077a5cf1471d" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 46803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:12:23.311092 517 consensus_queue.cc:1035] T 93c25f9445384aa2b2faa810f304bf94 P 6ae33e71c7d2437abcb65ed59923573d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d2bdbac60244a5d9a356958db1bb1c7" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:12:23.361912 32413 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34074:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:12:23.362186 32413 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34074:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:12:23.365016 32413 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3838eedbbb89456096bd20aa4891ea73: Sending DeleteTablet for 3 replicas of tablet 93c25f9445384aa2b2faa810f304bf94
I20260924 16:12:23.365903 32555 tablet_service.cc:1514] Processing DeleteTablet for tablet 93c25f9445384aa2b2faa810f304bf94 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:12:23 UTC) from {username='kudu', principal='kudu/127.29.234.62@KRBTEST.COM'} at 127.0.0.1:39246
I20260924 16:12:23.366008 32697 tablet_service.cc:1514] Processing DeleteTablet for tablet 93c25f9445384aa2b2faa810f304bf94 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:12:23 UTC) from {username='kudu', principal='kudu/127.29.234.62@KRBTEST.COM'} at 127.0.0.1:43082
I20260924 16:12:23.366031 369 tablet_service.cc:1514] Processing DeleteTablet for tablet 93c25f9445384aa2b2faa810f304bf94 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:12:23 UTC) from {username='kudu', principal='kudu/127.29.234.62@KRBTEST.COM'} at 127.0.0.1:37924
I20260924 16:12:23.366218 560 tablet_replica.cc:330] T 93c25f9445384aa2b2faa810f304bf94 P 445ada5dbd2542b7a204077a5cf1471d: stopping tablet replica
I20260924 16:12:23.366299 561 tablet_replica.cc:330] T 93c25f9445384aa2b2faa810f304bf94 P 6ae33e71c7d2437abcb65ed59923573d: stopping tablet replica
I20260924 16:12:23.366318 562 tablet_replica.cc:330] T 93c25f9445384aa2b2faa810f304bf94 P 9d2bdbac60244a5d9a356958db1bb1c7: stopping tablet replica
I20260924 16:12:23.366433 560 raft_consensus.cc:2229] T 93c25f9445384aa2b2faa810f304bf94 P 445ada5dbd2542b7a204077a5cf1471d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:12:23.366472 562 raft_consensus.cc:2229] T 93c25f9445384aa2b2faa810f304bf94 P 9d2bdbac60244a5d9a356958db1bb1c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:12:23.366474 561 raft_consensus.cc:2229] T 93c25f9445384aa2b2faa810f304bf94 P 6ae33e71c7d2437abcb65ed59923573d [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:12:23.366614 562 raft_consensus.cc:2258] T 93c25f9445384aa2b2faa810f304bf94 P 9d2bdbac60244a5d9a356958db1bb1c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:12:23.366626 560 raft_consensus.cc:2258] T 93c25f9445384aa2b2faa810f304bf94 P 445ada5dbd2542b7a204077a5cf1471d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:12:23.366786 561 raft_consensus.cc:2258] T 93c25f9445384aa2b2faa810f304bf94 P 6ae33e71c7d2437abcb65ed59923573d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:12:23.367772 562 ts_tablet_manager.cc:1905] T 93c25f9445384aa2b2faa810f304bf94 P 9d2bdbac60244a5d9a356958db1bb1c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:12:23.367903 561 ts_tablet_manager.cc:1905] T 93c25f9445384aa2b2faa810f304bf94 P 6ae33e71c7d2437abcb65ed59923573d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:12:23.368332 560 ts_tablet_manager.cc:1905] T 93c25f9445384aa2b2faa810f304bf94 P 445ada5dbd2542b7a204077a5cf1471d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:12:23.373071 560 ts_tablet_manager.cc:1918] T 93c25f9445384aa2b2faa810f304bf94 P 445ada5dbd2542b7a204077a5cf1471d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 16:12:23.373327 560 log.cc:1196] T 93c25f9445384aa2b2faa810f304bf94 P 445ada5dbd2542b7a204077a5cf1471d: Deleting WAL directory at /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/minicluster-data/ts-0/wal/wals/93c25f9445384aa2b2faa810f304bf94
I20260924 16:12:23.373274 32413 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34074:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 16:12:23.373863 560 ts_tablet_manager.cc:1939] T 93c25f9445384aa2b2faa810f304bf94 P 445ada5dbd2542b7a204077a5cf1471d: Deleting consensus metadata
W20260924 16:12:23.374251 32413 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:12:23.374516 562 ts_tablet_manager.cc:1918] T 93c25f9445384aa2b2faa810f304bf94 P 9d2bdbac60244a5d9a356958db1bb1c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 16:12:23.374699 562 log.cc:1196] T 93c25f9445384aa2b2faa810f304bf94 P 9d2bdbac60244a5d9a356958db1bb1c7: Deleting WAL directory at /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/minicluster-data/ts-1/wal/wals/93c25f9445384aa2b2faa810f304bf94
I20260924 16:12:23.375152 562 ts_tablet_manager.cc:1939] T 93c25f9445384aa2b2faa810f304bf94 P 9d2bdbac60244a5d9a356958db1bb1c7: Deleting consensus metadata
I20260924 16:12:23.375170 32400 catalog_manager.cc:4841] TS 445ada5dbd2542b7a204077a5cf1471d (127.29.234.1:46803): tablet 93c25f9445384aa2b2faa810f304bf94 (table test-table [id=c78e2e40dc87415191fd102b6a816dd1]) successfully deleted
I20260924 16:12:23.375180 561 ts_tablet_manager.cc:1918] T 93c25f9445384aa2b2faa810f304bf94 P 6ae33e71c7d2437abcb65ed59923573d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 16:12:23.375349 561 log.cc:1196] T 93c25f9445384aa2b2faa810f304bf94 P 6ae33e71c7d2437abcb65ed59923573d: Deleting WAL directory at /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/minicluster-data/ts-2/wal/wals/93c25f9445384aa2b2faa810f304bf94
I20260924 16:12:23.375895 561 ts_tablet_manager.cc:1939] T 93c25f9445384aa2b2faa810f304bf94 P 6ae33e71c7d2437abcb65ed59923573d: Deleting consensus metadata
I20260924 16:12:23.376317 32401 catalog_manager.cc:4841] TS 9d2bdbac60244a5d9a356958db1bb1c7 (127.29.234.2:41705): tablet 93c25f9445384aa2b2faa810f304bf94 (table test-table [id=c78e2e40dc87415191fd102b6a816dd1]) successfully deleted
I20260924 16:12:23.377512 32400 catalog_manager.cc:4841] TS 6ae33e71c7d2437abcb65ed59923573d (127.29.234.3:42009): tablet 93c25f9445384aa2b2faa810f304bf94 (table test-table [id=c78e2e40dc87415191fd102b6a816dd1]) successfully deleted
I20260924 16:12:23.381726 32555 tablet_service.cc:1467] Processing CreateTablet for tablet b50ff3b7c8ce4d6bbb43a2785bfa5b3d (DEFAULT_TABLE table=test-table [id=2f360a166f4045a88dab7f780f5f5187]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:12:23.381927 32697 tablet_service.cc:1467] Processing CreateTablet for tablet b50ff3b7c8ce4d6bbb43a2785bfa5b3d (DEFAULT_TABLE table=test-table [id=2f360a166f4045a88dab7f780f5f5187]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:12:23.382175 32555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b50ff3b7c8ce4d6bbb43a2785bfa5b3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:23.382391 32697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b50ff3b7c8ce4d6bbb43a2785bfa5b3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:23.387868 571 tablet_bootstrap.cc:492] T b50ff3b7c8ce4d6bbb43a2785bfa5b3d P 9d2bdbac60244a5d9a356958db1bb1c7: Bootstrap starting.
I20260924 16:12:23.387969 369 tablet_service.cc:1467] Processing CreateTablet for tablet b50ff3b7c8ce4d6bbb43a2785bfa5b3d (DEFAULT_TABLE table=test-table [id=2f360a166f4045a88dab7f780f5f5187]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:12:23.388432 369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b50ff3b7c8ce4d6bbb43a2785bfa5b3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:23.389382 572 tablet_bootstrap.cc:492] T b50ff3b7c8ce4d6bbb43a2785bfa5b3d P 445ada5dbd2542b7a204077a5cf1471d: Bootstrap starting.
I20260924 16:12:23.389318 571 tablet_bootstrap.cc:654] T b50ff3b7c8ce4d6bbb43a2785bfa5b3d P 9d2bdbac60244a5d9a356958db1bb1c7: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:23.391124 572 tablet_bootstrap.cc:654] T b50ff3b7c8ce4d6bbb43a2785bfa5b3d P 445ada5dbd2542b7a204077a5cf1471d: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:23.391820 571 tablet_bootstrap.cc:492] T b50ff3b7c8ce4d6bbb43a2785bfa5b3d P 9d2bdbac60244a5d9a356958db1bb1c7: No bootstrap required, opened a new log
I20260924 16:12:23.391992 571 ts_tablet_manager.cc:1397] T b50ff3b7c8ce4d6bbb43a2785bfa5b3d P 9d2bdbac60244a5d9a356958db1bb1c7: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260924 16:12:23.392812 571 raft_consensus.cc:348] T b50ff3b7c8ce4d6bbb43a2785bfa5b3d P 9d2bdbac60244a5d9a356958db1bb1c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445ada5dbd2542b7a204077a5cf1471d" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 46803 } } peers { permanent_uuid: "9d2bdbac60244a5d9a356958db1bb1c7" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41705 } } peers { permanent_uuid: "6ae33e71c7d2437abcb65ed59923573d" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 42009 } }
I20260924 16:12:23.393056 571 raft_consensus.cc:374] T b50ff3b7c8ce4d6bbb43a2785bfa5b3d P 9d2bdbac60244a5d9a356958db1bb1c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:23.393133 571 raft_consensus.cc:729] T b50ff3b7c8ce4d6bbb43a2785bfa5b3d P 9d2bdbac60244a5d9a356958db1bb1c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d2bdbac60244a5d9a356958db1bb1c7, State: Initialized, Role: FOLLOWER
I20260924 16:12:23.393272 572 tablet_bootstrap.cc:492] T b50ff3b7c8ce4d6bbb43a2785bfa5b3d P 445ada5dbd2542b7a204077a5cf1471d: No bootstrap required, opened a new log
I20260924 16:12:23.393476 572 ts_tablet_manager.cc:1397] T b50ff3b7c8ce4d6bbb43a2785bfa5b3d P 445ada5dbd2542b7a204077a5cf1471d: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.003s
I20260924 16:12:23.393440 571 consensus_queue.cc:260] T b50ff3b7c8ce4d6bbb43a2785bfa5b3d P 9d2bdbac60244a5d9a356958db1bb1c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445ada5dbd2542b7a204077a5cf1471d" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 46803 } } peers { permanent_uuid: "9d2bdbac60244a5d9a356958db1bb1c7" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41705 } } peers { permanent_uuid: "6ae33e71c7d2437abcb65ed59923573d" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 42009 } }
I20260924 16:12:23.393993 571 ts_tablet_manager.cc:1428] T b50ff3b7c8ce4d6bbb43a2785bfa5b3d P 9d2bdbac60244a5d9a356958db1bb1c7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260924 16:12:23.394203 572 raft_consensus.cc:348] T b50ff3b7c8ce4d6bbb43a2785bfa5b3d P 445ada5dbd2542b7a204077a5cf1471d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445ada5dbd2542b7a204077a5cf1471d" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 46803 } } peers { permanent_uuid: "9d2bdbac60244a5d9a356958db1bb1c7" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41705 } } peers { permanent_uuid: "6ae33e71c7d2437abcb65ed59923573d" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 42009 } }
I20260924 16:12:23.394465 572 raft_consensus.cc:374] T b50ff3b7c8ce4d6bbb43a2785bfa5b3d P 445ada5dbd2542b7a204077a5cf1471d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:23.394584 572 raft_consensus.cc:729] T b50ff3b7c8ce4d6bbb43a2785bfa5b3d P 445ada5dbd2542b7a204077a5cf1471d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 445ada5dbd2542b7a204077a5cf1471d, State: Initialized, Role: FOLLOWER
I20260924 16:12:23.394801 573 tablet_bootstrap.cc:492] T b50ff3b7c8ce4d6bbb43a2785bfa5b3d P 6ae33e71c7d2437abcb65ed59923573d: Bootstrap starting.
I20260924 16:12:23.394838 572 consensus_queue.cc:260] T b50ff3b7c8ce4d6bbb43a2785bfa5b3d P 445ada5dbd2542b7a204077a5cf1471d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445ada5dbd2542b7a204077a5cf1471d" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 46803 } } peers { permanent_uuid: "9d2bdbac60244a5d9a356958db1bb1c7" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41705 } } peers { permanent_uuid: "6ae33e71c7d2437abcb65ed59923573d" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 42009 } }
I20260924 16:12:23.395524 572 ts_tablet_manager.cc:1428] T b50ff3b7c8ce4d6bbb43a2785bfa5b3d P 445ada5dbd2542b7a204077a5cf1471d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260924 16:12:23.396473 573 tablet_bootstrap.cc:654] T b50ff3b7c8ce4d6bbb43a2785bfa5b3d P 6ae33e71c7d2437abcb65ed59923573d: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:23.398559 573 tablet_bootstrap.cc:492] T b50ff3b7c8ce4d6bbb43a2785bfa5b3d P 6ae33e71c7d2437abcb65ed59923573d: No bootstrap required, opened a new log
I20260924 16:12:23.398780 573 ts_tablet_manager.cc:1397] T b50ff3b7c8ce4d6bbb43a2785bfa5b3d P 6ae33e71c7d2437abcb65ed59923573d: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260924 16:12:23.399351 573 raft_consensus.cc:348] T b50ff3b7c8ce4d6bbb43a2785bfa5b3d P 6ae33e71c7d2437abcb65ed59923573d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445ada5dbd2542b7a204077a5cf1471d" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 46803 } } peers { permanent_uuid: "9d2bdbac60244a5d9a356958db1bb1c7" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41705 } } peers { permanent_uuid: "6ae33e71c7d2437abcb65ed59923573d" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 42009 } }
I20260924 16:12:23.399603 573 raft_consensus.cc:374] T b50ff3b7c8ce4d6bbb43a2785bfa5b3d P 6ae33e71c7d2437abcb65ed59923573d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:23.399681 573 raft_consensus.cc:729] T b50ff3b7c8ce4d6bbb43a2785bfa5b3d P 6ae33e71c7d2437abcb65ed59923573d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ae33e71c7d2437abcb65ed59923573d, State: Initialized, Role: FOLLOWER
I20260924 16:12:23.399947 573 consensus_queue.cc:260] T b50ff3b7c8ce4d6bbb43a2785bfa5b3d P 6ae33e71c7d2437abcb65ed59923573d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445ada5dbd2542b7a204077a5cf1471d" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 46803 } } peers { permanent_uuid: "9d2bdbac60244a5d9a356958db1bb1c7" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41705 } } peers { permanent_uuid: "6ae33e71c7d2437abcb65ed59923573d" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 42009 } }
I20260924 16:12:23.401175 573 ts_tablet_manager.cc:1428] T b50ff3b7c8ce4d6bbb43a2785bfa5b3d P 6ae33e71c7d2437abcb65ed59923573d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 16:12:23.535454 551 raft_consensus.cc:482] T b50ff3b7c8ce4d6bbb43a2785bfa5b3d P 445ada5dbd2542b7a204077a5cf1471d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:12:23.535676 551 raft_consensus.cc:504] T b50ff3b7c8ce4d6bbb43a2785bfa5b3d P 445ada5dbd2542b7a204077a5cf1471d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445ada5dbd2542b7a204077a5cf1471d" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 46803 } } peers { permanent_uuid: "9d2bdbac60244a5d9a356958db1bb1c7" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41705 } } peers { permanent_uuid: "6ae33e71c7d2437abcb65ed59923573d" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 42009 } }
I20260924 16:12:23.536489 551 leader_election.cc:288] T b50ff3b7c8ce4d6bbb43a2785bfa5b3d P 445ada5dbd2542b7a204077a5cf1471d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d2bdbac60244a5d9a356958db1bb1c7 (127.29.234.2:41705), 6ae33e71c7d2437abcb65ed59923573d (127.29.234.3:42009)
I20260924 16:12:23.537177 32717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b50ff3b7c8ce4d6bbb43a2785bfa5b3d" candidate_uuid: "445ada5dbd2542b7a204077a5cf1471d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d2bdbac60244a5d9a356958db1bb1c7" is_pre_election: true
I20260924 16:12:23.537202 396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b50ff3b7c8ce4d6bbb43a2785bfa5b3d" candidate_uuid: "445ada5dbd2542b7a204077a5cf1471d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ae33e71c7d2437abcb65ed59923573d" is_pre_election: true
I20260924 16:12:23.537508 32717 raft_consensus.cc:2454] T b50ff3b7c8ce4d6bbb43a2785bfa5b3d P 9d2bdbac60244a5d9a356958db1bb1c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 445ada5dbd2542b7a204077a5cf1471d in term 0.
I20260924 16:12:23.537551 396 raft_consensus.cc:2454] T b50ff3b7c8ce4d6bbb43a2785bfa5b3d P 6ae33e71c7d2437abcb65ed59923573d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 445ada5dbd2542b7a204077a5cf1471d in term 0.
I20260924 16:12:23.538018 32506 leader_election.cc:302] T b50ff3b7c8ce4d6bbb43a2785bfa5b3d P 445ada5dbd2542b7a204077a5cf1471d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 445ada5dbd2542b7a204077a5cf1471d, 6ae33e71c7d2437abcb65ed59923573d, 9d2bdbac60244a5d9a356958db1bb1c7; no voters:
I20260924 16:12:23.538334 551 raft_consensus.cc:2781] T b50ff3b7c8ce4d6bbb43a2785bfa5b3d P 445ada5dbd2542b7a204077a5cf1471d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:12:23.538466 551 raft_consensus.cc:482] T b50ff3b7c8ce4d6bbb43a2785bfa5b3d P 445ada5dbd2542b7a204077a5cf1471d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:12:23.538579 551 raft_consensus.cc:3037] T b50ff3b7c8ce4d6bbb43a2785bfa5b3d P 445ada5dbd2542b7a204077a5cf1471d [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:23.541038 551 raft_consensus.cc:504] T b50ff3b7c8ce4d6bbb43a2785bfa5b3d P 445ada5dbd2542b7a204077a5cf1471d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445ada5dbd2542b7a204077a5cf1471d" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 46803 } } peers { permanent_uuid: "9d2bdbac60244a5d9a356958db1bb1c7" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41705 } } peers { permanent_uuid: "6ae33e71c7d2437abcb65ed59923573d" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 42009 } }
I20260924 16:12:23.541946 551 leader_election.cc:288] T b50ff3b7c8ce4d6bbb43a2785bfa5b3d P 445ada5dbd2542b7a204077a5cf1471d [CANDIDATE]: Term 1 election: Requested vote from peers 9d2bdbac60244a5d9a356958db1bb1c7 (127.29.234.2:41705), 6ae33e71c7d2437abcb65ed59923573d (127.29.234.3:42009)
I20260924 16:12:23.542310 396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b50ff3b7c8ce4d6bbb43a2785bfa5b3d" candidate_uuid: "445ada5dbd2542b7a204077a5cf1471d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ae33e71c7d2437abcb65ed59923573d"
I20260924 16:12:23.542320 32717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b50ff3b7c8ce4d6bbb43a2785bfa5b3d" candidate_uuid: "445ada5dbd2542b7a204077a5cf1471d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d2bdbac60244a5d9a356958db1bb1c7"
I20260924 16:12:23.542541 396 raft_consensus.cc:3037] T b50ff3b7c8ce4d6bbb43a2785bfa5b3d P 6ae33e71c7d2437abcb65ed59923573d [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:23.542584 32717 raft_consensus.cc:3037] T b50ff3b7c8ce4d6bbb43a2785bfa5b3d P 9d2bdbac60244a5d9a356958db1bb1c7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:23.544620 396 raft_consensus.cc:2454] T b50ff3b7c8ce4d6bbb43a2785bfa5b3d P 6ae33e71c7d2437abcb65ed59923573d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 445ada5dbd2542b7a204077a5cf1471d in term 1.
I20260924 16:12:23.544796 32717 raft_consensus.cc:2454] T b50ff3b7c8ce4d6bbb43a2785bfa5b3d P 9d2bdbac60244a5d9a356958db1bb1c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 445ada5dbd2542b7a204077a5cf1471d in term 1.
I20260924 16:12:23.545110 32505 leader_election.cc:302] T b50ff3b7c8ce4d6bbb43a2785bfa5b3d P 445ada5dbd2542b7a204077a5cf1471d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 445ada5dbd2542b7a204077a5cf1471d, 6ae33e71c7d2437abcb65ed59923573d; no voters:
I20260924 16:12:23.545625 551 raft_consensus.cc:2781] T b50ff3b7c8ce4d6bbb43a2785bfa5b3d P 445ada5dbd2542b7a204077a5cf1471d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:12:23.546098 551 raft_consensus.cc:686] T b50ff3b7c8ce4d6bbb43a2785bfa5b3d P 445ada5dbd2542b7a204077a5cf1471d [term 1 LEADER]: Becoming Leader. State: Replica: 445ada5dbd2542b7a204077a5cf1471d, State: Running, Role: LEADER
I20260924 16:12:23.546473 551 consensus_queue.cc:237] T b50ff3b7c8ce4d6bbb43a2785bfa5b3d P 445ada5dbd2542b7a204077a5cf1471d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445ada5dbd2542b7a204077a5cf1471d" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 46803 } } peers { permanent_uuid: "9d2bdbac60244a5d9a356958db1bb1c7" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41705 } } peers { permanent_uuid: "6ae33e71c7d2437abcb65ed59923573d" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 42009 } }
I20260924 16:12:23.549172 32413 catalog_manager.cc:5495] T b50ff3b7c8ce4d6bbb43a2785bfa5b3d P 445ada5dbd2542b7a204077a5cf1471d reported cstate change: term changed from 0 to 1, leader changed from <none> to 445ada5dbd2542b7a204077a5cf1471d (127.29.234.1). New cstate: current_term: 1 leader_uuid: "445ada5dbd2542b7a204077a5cf1471d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445ada5dbd2542b7a204077a5cf1471d" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 46803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d2bdbac60244a5d9a356958db1bb1c7" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ae33e71c7d2437abcb65ed59923573d" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 42009 } health_report { overall_health: UNKNOWN } } }
I20260924 16:12:23.596537 592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:23.591315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50108 (local address 127.29.234.1:46803)
0924 16:12:23.591841 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:23.591846 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:23.591867 (+ 21us) server_negotiation.cc:407] Connection header received
0924 16:12:23.591919 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:23.591923 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:23.591981 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:23.592061 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:12:23.592499 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:23.593228 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:23.594714 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:23.595127 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:23.595293 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:23.595781 (+ 488us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:12:23.595901 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:12:23.596003 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:23.596149 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 16:12:23.596228 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":287,"threads_started":1}
I20260924 16:12:23.606340 396 raft_consensus.cc:1261] T b50ff3b7c8ce4d6bbb43a2785bfa5b3d P 6ae33e71c7d2437abcb65ed59923573d [term 1 FOLLOWER]: Refusing update from remote peer 445ada5dbd2542b7a204077a5cf1471d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:12:23.606490 32717 raft_consensus.cc:1261] T b50ff3b7c8ce4d6bbb43a2785bfa5b3d P 9d2bdbac60244a5d9a356958db1bb1c7 [term 1 FOLLOWER]: Refusing update from remote peer 445ada5dbd2542b7a204077a5cf1471d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:12:23.606986 589 consensus_queue.cc:1035] T b50ff3b7c8ce4d6bbb43a2785bfa5b3d P 445ada5dbd2542b7a204077a5cf1471d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ae33e71c7d2437abcb65ed59923573d" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 42009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:12:23.607344 551 consensus_queue.cc:1035] T b50ff3b7c8ce4d6bbb43a2785bfa5b3d P 445ada5dbd2542b7a204077a5cf1471d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d2bdbac60244a5d9a356958db1bb1c7" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 16:12:23.630389 542 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 16:12:25.210561 32415 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34074:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:12:25.210798 32415 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34074:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:12:25.214956 32415 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3838eedbbb89456096bd20aa4891ea73: Sending DeleteTablet for 3 replicas of tablet b50ff3b7c8ce4d6bbb43a2785bfa5b3d
I20260924 16:12:25.215763 32555 tablet_service.cc:1514] Processing DeleteTablet for tablet b50ff3b7c8ce4d6bbb43a2785bfa5b3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:12:25 UTC) from {username='kudu', principal='kudu/127.29.234.62@KRBTEST.COM'} at 127.0.0.1:39246
I20260924 16:12:25.215865 369 tablet_service.cc:1514] Processing DeleteTablet for tablet b50ff3b7c8ce4d6bbb43a2785bfa5b3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:12:25 UTC) from {username='kudu', principal='kudu/127.29.234.62@KRBTEST.COM'} at 127.0.0.1:37924
I20260924 16:12:25.215840 32697 tablet_service.cc:1514] Processing DeleteTablet for tablet b50ff3b7c8ce4d6bbb43a2785bfa5b3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:12:25 UTC) from {username='kudu', principal='kudu/127.29.234.62@KRBTEST.COM'} at 127.0.0.1:43082
I20260924 16:12:25.216590 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 32487
I20260924 16:12:25.225898 621 tablet_replica.cc:330] T b50ff3b7c8ce4d6bbb43a2785bfa5b3d P 9d2bdbac60244a5d9a356958db1bb1c7: stopping tablet replica
I20260924 16:12:25.226194 621 raft_consensus.cc:2229] T b50ff3b7c8ce4d6bbb43a2785bfa5b3d P 9d2bdbac60244a5d9a356958db1bb1c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:12:25.226511 621 raft_consensus.cc:2258] T b50ff3b7c8ce4d6bbb43a2785bfa5b3d P 9d2bdbac60244a5d9a356958db1bb1c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:12:25.227118 619 tablet_replica.cc:330] T b50ff3b7c8ce4d6bbb43a2785bfa5b3d P 445ada5dbd2542b7a204077a5cf1471d: stopping tablet replica
I20260924 16:12:25.227377 619 raft_consensus.cc:2229] T b50ff3b7c8ce4d6bbb43a2785bfa5b3d P 445ada5dbd2542b7a204077a5cf1471d [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:12:25.227857 619 raft_consensus.cc:2258] T b50ff3b7c8ce4d6bbb43a2785bfa5b3d P 445ada5dbd2542b7a204077a5cf1471d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:12:25.227907 621 ts_tablet_manager.cc:1905] T b50ff3b7c8ce4d6bbb43a2785bfa5b3d P 9d2bdbac60244a5d9a356958db1bb1c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:12:25.229278 619 ts_tablet_manager.cc:1905] T b50ff3b7c8ce4d6bbb43a2785bfa5b3d P 445ada5dbd2542b7a204077a5cf1471d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:12:25.229403 620 tablet_replica.cc:330] T b50ff3b7c8ce4d6bbb43a2785bfa5b3d P 6ae33e71c7d2437abcb65ed59923573d: stopping tablet replica
I20260924 16:12:25.229681 620 raft_consensus.cc:2229] T b50ff3b7c8ce4d6bbb43a2785bfa5b3d P 6ae33e71c7d2437abcb65ed59923573d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:12:25.229982 620 raft_consensus.cc:2258] T b50ff3b7c8ce4d6bbb43a2785bfa5b3d P 6ae33e71c7d2437abcb65ed59923573d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:12:25.230994 620 ts_tablet_manager.cc:1905] T b50ff3b7c8ce4d6bbb43a2785bfa5b3d P 6ae33e71c7d2437abcb65ed59923573d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:12:25.234920 621 ts_tablet_manager.cc:1918] T b50ff3b7c8ce4d6bbb43a2785bfa5b3d P 9d2bdbac60244a5d9a356958db1bb1c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 16:12:25.235144 621 log.cc:1196] T b50ff3b7c8ce4d6bbb43a2785bfa5b3d P 9d2bdbac60244a5d9a356958db1bb1c7: Deleting WAL directory at /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/minicluster-data/ts-1/wal/wals/b50ff3b7c8ce4d6bbb43a2785bfa5b3d
I20260924 16:12:25.235677 621 ts_tablet_manager.cc:1939] T b50ff3b7c8ce4d6bbb43a2785bfa5b3d P 9d2bdbac60244a5d9a356958db1bb1c7: Deleting consensus metadata
I20260924 16:12:25.236471 619 ts_tablet_manager.cc:1918] T b50ff3b7c8ce4d6bbb43a2785bfa5b3d P 445ada5dbd2542b7a204077a5cf1471d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 16:12:25.236683 619 log.cc:1196] T b50ff3b7c8ce4d6bbb43a2785bfa5b3d P 445ada5dbd2542b7a204077a5cf1471d: Deleting WAL directory at /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/minicluster-data/ts-0/wal/wals/b50ff3b7c8ce4d6bbb43a2785bfa5b3d
I20260924 16:12:25.236943 32401 catalog_manager.cc:4841] TS 9d2bdbac60244a5d9a356958db1bb1c7 (127.29.234.2:41705): tablet b50ff3b7c8ce4d6bbb43a2785bfa5b3d (table test-table [id=2f360a166f4045a88dab7f780f5f5187]) successfully deleted
I20260924 16:12:25.237308 619 ts_tablet_manager.cc:1939] T b50ff3b7c8ce4d6bbb43a2785bfa5b3d P 445ada5dbd2542b7a204077a5cf1471d: Deleting consensus metadata
I20260924 16:12:25.237546 620 ts_tablet_manager.cc:1918] T b50ff3b7c8ce4d6bbb43a2785bfa5b3d P 6ae33e71c7d2437abcb65ed59923573d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 16:12:25.237699 620 log.cc:1196] T b50ff3b7c8ce4d6bbb43a2785bfa5b3d P 6ae33e71c7d2437abcb65ed59923573d: Deleting WAL directory at /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790266323169505-30632-0/minicluster-data/ts-2/wal/wals/b50ff3b7c8ce4d6bbb43a2785bfa5b3d
I20260924 16:12:25.238104 620 ts_tablet_manager.cc:1939] T b50ff3b7c8ce4d6bbb43a2785bfa5b3d P 6ae33e71c7d2437abcb65ed59923573d: Deleting consensus metadata
I20260924 16:12:25.238680 32400 catalog_manager.cc:4841] TS 445ada5dbd2542b7a204077a5cf1471d (127.29.234.1:46803): tablet b50ff3b7c8ce4d6bbb43a2785bfa5b3d (table test-table [id=2f360a166f4045a88dab7f780f5f5187]) successfully deleted
I20260924 16:12:25.239089 32400 catalog_manager.cc:4841] TS 6ae33e71c7d2437abcb65ed59923573d (127.29.234.3:42009): tablet b50ff3b7c8ce4d6bbb43a2785bfa5b3d (table test-table [id=2f360a166f4045a88dab7f780f5f5187]) successfully deleted
Sep 24 16:12:25 dist-test-slave-v5j9 krb5kdc[32370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266343, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.29.234.1@KRBTEST.COM
I20260924 16:12:25.246695 623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:25.218245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50110 (local address 127.29.234.1:46803)
0924 16:12:25.228486 (+ 10241us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:25.228494 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:25.228526 (+ 32us) server_negotiation.cc:407] Connection header received
0924 16:12:25.228591 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:25.228612 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:25.228699 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:25.228834 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:25.232181 (+ 3347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:25.233315 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:25.235264 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:25.235922 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:25.236016 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:25.241103 (+ 5087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:25.241118 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:25.241137 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:25.241168 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:25.244208 (+ 3040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:25.244835 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:25.244842 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:25.244844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:25.244976 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:25.245328 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:25.245331 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:25.245333 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:25.245749 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:25.245968 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:25.246265 (+ 297us) server_negotiation.cc:299] Negotiation successful
0924 16:12:25.246441 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10077,"thread_start_us":9857,"threads_started":1}
W20260924 16:12:25.247401 32622 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50110
W20260924 16:12:25.248024 30632 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 32487: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 16:12:25.248270 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 32487
I20260924 16:12:25.271934 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 32631
Sep 24 16:12:25 dist-test-slave-v5j9 krb5kdc[32370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266343, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.29.234.2@KRBTEST.COM
I20260924 16:12:25.294447 624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:25.272788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46160 (local address 127.29.234.2:41705)
0924 16:12:25.273410 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:25.273415 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:25.273431 (+ 16us) server_negotiation.cc:407] Connection header received
0924 16:12:25.273484 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:25.273488 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:25.273551 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:25.273640 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:25.274933 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:25.275662 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:25.276912 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:25.277313 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:25.277548 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:25.288787 (+ 11239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:25.288804 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:25.288825 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:25.288876 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:25.291724 (+ 2848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:25.292396 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:25.292404 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:25.292407 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:25.292546 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:25.292895 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:25.292898 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:25.292900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:25.293234 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:25.293519 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:25.293833 (+ 314us) server_negotiation.cc:299] Negotiation successful
0924 16:12:25.294011 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":275,"threads_started":1}
W20260924 16:12:25.295147 32763 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46160
W20260924 16:12:25.295849 30632 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 32631: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 16:12:25.296008 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 32631
I20260924 16:12:25.313848 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 304
Sep 24 16:12:25 dist-test-slave-v5j9 krb5kdc[32370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266343, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.29.234.3@KRBTEST.COM
I20260924 16:12:25.330572 625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:25.314705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42950 (local address 127.29.234.3:42009)
0924 16:12:25.315231 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:25.315236 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:25.315252 (+ 16us) server_negotiation.cc:407] Connection header received
0924 16:12:25.315308 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:25.315312 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:25.315376 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:25.315467 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:25.316703 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:25.317536 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:25.318739 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:25.319159 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:25.319266 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:25.324370 (+ 5104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:25.324386 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:25.324405 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:25.324440 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:25.327610 (+ 3170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:25.328334 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:25.328342 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:25.328345 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:25.328479 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:25.328870 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:25.328875 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:25.328878 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:25.329305 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:25.329637 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:25.329968 (+ 331us) server_negotiation.cc:299] Negotiation successful
0924 16:12:25.330217 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":284,"threads_started":1}
W20260924 16:12:25.331295 437 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42950
W20260924 16:12:25.332010 30632 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 304: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 16:12:25.332211 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 304
I20260924 16:12:25.360836 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 32383
I20260924 16:12:25.362076 32456 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:12:25.478995 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 32383
2026-09-24T16:12:25Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8879 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:12:25 dist-test-slave-v5j9 krb5kdc[629](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:12:25 dist-test-slave-v5j9 krb5kdc[629](info): set up 2 sockets
Sep 24 16:12:25 dist-test-slave-v5j9 krb5kdc[629](info): commencing operation
krb5kdc: starting...
W20260924 16:12:26.580816 30632 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Sep 24 16:12:26 dist-test-slave-v5j9 krb5kdc[629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266346, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:12:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:12:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.234.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab.
Entry for principal kudu/127.29.234.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab.
I20260924 16:12:26.775095 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:44055
--webserver_interface=127.29.234.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab
--principal=kudu/127.29.234.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:39223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.234.62:44055
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:12:27.106271 642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:27.106649 642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:27.106822 642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:27.114010 642 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:12:27.114173 642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:27.114288 642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:27.114403 642 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:12:27.114494 642 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:12:27.124320 642 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:39223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.234.62:44055
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab
--principal=kudu/127.29.234.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:44055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.234.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:12:27.125730 642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:27.127120 642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:27.139987 648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:27.139987 650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:27.139958 647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:27.140830 642 server_base.cc:1034] running on GCE node
I20260924 16:12:27.141696 642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:27.143178 642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:27.144434 642 hybrid_clock.cc:648] HybridClock initialized: now 1790266347144392 us; error 39 us; skew 500 ppm
I20260924 16:12:27.144804 642 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:12:27 dist-test-slave-v5j9 krb5kdc[629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266347, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:27.147993 642 init.cc:377] Logged in from keytab as kudu/127.29.234.62@KRBTEST.COM (short username kudu)
I20260924 16:12:27.149667 642 webserver.cc:460] Webserver started at http://127.29.234.62:33623/ using document root <none> and password file <none>
I20260924 16:12:27.150406 642 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:27.150556 642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:27.150887 642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:27.154132 642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/minicluster-data/master-0/data/instance:
uuid: "bd85ef13bddc4c30942715ac113968d6"
format_stamp: "Formatted at 2026-09-24 16:12:27 on dist-test-slave-v5j9"
server_key: "0c19cd9ee01f474c8e685ff8476f05e0"
server_key_iv: "673b683679c9b64c27ce78f92333e2dc"
server_key_version: "encryptionkey@0"
I20260924 16:12:27.154944 642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/minicluster-data/master-0/wal/instance:
uuid: "bd85ef13bddc4c30942715ac113968d6"
format_stamp: "Formatted at 2026-09-24 16:12:27 on dist-test-slave-v5j9"
server_key: "0c19cd9ee01f474c8e685ff8476f05e0"
server_key_iv: "673b683679c9b64c27ce78f92333e2dc"
server_key_version: "encryptionkey@0"
I20260924 16:12:27.160802 642 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260924 16:12:27.164784 657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:27.166359 642 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:12:27.166545 642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/minicluster-data/master-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/minicluster-data/master-0/wal
uuid: "bd85ef13bddc4c30942715ac113968d6"
format_stamp: "Formatted at 2026-09-24 16:12:27 on dist-test-slave-v5j9"
server_key: "0c19cd9ee01f474c8e685ff8476f05e0"
server_key_iv: "673b683679c9b64c27ce78f92333e2dc"
server_key_version: "encryptionkey@0"
I20260924 16:12:27.166761 642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:27.193539 642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:27.197482 642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:27.197779 642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:27.216295 642 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.62:44055
I20260924 16:12:27.216303 710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.62:44055 every 8 connection(s)
I20260924 16:12:27.217760 642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
I20260924 16:12:27.221586 711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:27.224314 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 642
I20260924 16:12:27.224571 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/minicluster-data/master-0/wal/instance
I20260924 16:12:27.225378 30632 external_mini_cluster.cc:1428] Setting key 2633e7b4ca356d66a44275d26d452fca
I20260924 16:12:27.234352 711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd85ef13bddc4c30942715ac113968d6: Bootstrap starting.
I20260924 16:12:27.238204 711 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd85ef13bddc4c30942715ac113968d6: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:27.239564 711 log.cc:824] T 00000000000000000000000000000000 P bd85ef13bddc4c30942715ac113968d6: Log is configured to *not* fsync() on all Append() calls
Sep 24 16:12:27 dist-test-slave-v5j9 krb5kdc[629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266346, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:12:27.243007 711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd85ef13bddc4c30942715ac113968d6: No bootstrap required, opened a new log
I20260924 16:12:27.250190 714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:27.227492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39928 (local address 127.29.234.62:44055)
0924 16:12:27.228289 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:27.228300 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:27.228341 (+ 41us) server_negotiation.cc:407] Connection header received
0924 16:12:27.230070 (+ 1729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:27.230094 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:27.230498 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:27.230841 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:27.232598 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:27.234021 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:27.235751 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:27.236451 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:27.236588 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:27.241570 (+ 4982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:27.241581 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:27.241600 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:27.241648 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:27.245545 (+ 3897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:27.246220 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:27.246236 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:27.246248 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:27.246461 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:27.246812 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:27.246817 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:27.246820 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:27.247566 (+ 746us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:27.247852 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:27.248231 (+ 379us) server_negotiation.cc:299] Negotiation successful
0924 16:12:27.248714 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":370,"threads_started":1}
I20260924 16:12:27.253063 711 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd85ef13bddc4c30942715ac113968d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd85ef13bddc4c30942715ac113968d6" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 44055 } }
I20260924 16:12:27.253456 711 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd85ef13bddc4c30942715ac113968d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:27.253541 711 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd85ef13bddc4c30942715ac113968d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd85ef13bddc4c30942715ac113968d6, State: Initialized, Role: FOLLOWER
I20260924 16:12:27.254029 711 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd85ef13bddc4c30942715ac113968d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd85ef13bddc4c30942715ac113968d6" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 44055 } }
I20260924 16:12:27.254240 711 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd85ef13bddc4c30942715ac113968d6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:12:27.254395 711 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd85ef13bddc4c30942715ac113968d6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:12:27.254577 711 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd85ef13bddc4c30942715ac113968d6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:27.256702 711 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd85ef13bddc4c30942715ac113968d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd85ef13bddc4c30942715ac113968d6" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 44055 } }
I20260924 16:12:27.257208 711 leader_election.cc:302] T 00000000000000000000000000000000 P bd85ef13bddc4c30942715ac113968d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd85ef13bddc4c30942715ac113968d6; no voters:
I20260924 16:12:27.257813 711 leader_election.cc:288] T 00000000000000000000000000000000 P bd85ef13bddc4c30942715ac113968d6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:12:27.257965 716 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd85ef13bddc4c30942715ac113968d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:12:27.258639 716 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd85ef13bddc4c30942715ac113968d6 [term 1 LEADER]: Becoming Leader. State: Replica: bd85ef13bddc4c30942715ac113968d6, State: Running, Role: LEADER
I20260924 16:12:27.259187 716 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd85ef13bddc4c30942715ac113968d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd85ef13bddc4c30942715ac113968d6" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 44055 } }
I20260924 16:12:27.259948 711 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd85ef13bddc4c30942715ac113968d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:12:27.261996 718 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd85ef13bddc4c30942715ac113968d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd85ef13bddc4c30942715ac113968d6. Latest consensus state: current_term: 1 leader_uuid: "bd85ef13bddc4c30942715ac113968d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd85ef13bddc4c30942715ac113968d6" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 44055 } } }
I20260924 16:12:27.262364 718 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd85ef13bddc4c30942715ac113968d6 [sys.catalog]: This master's current role is: LEADER
I20260924 16:12:27.263521 725 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:12:27.265379 717 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd85ef13bddc4c30942715ac113968d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd85ef13bddc4c30942715ac113968d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd85ef13bddc4c30942715ac113968d6" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 44055 } } }
I20260924 16:12:27.265811 717 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd85ef13bddc4c30942715ac113968d6 [sys.catalog]: This master's current role is: LEADER
I20260924 16:12:27.271854 725 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:12:27.281658 725 catalog_manager.cc:1347] Generated new cluster ID: 73ad7414d9d443c58b907f5e513f9995
I20260924 16:12:27.281808 725 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:12:27.293774 725 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:12:27.295113 725 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:12:27.303740 725 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd85ef13bddc4c30942715ac113968d6: Generated new TSK 0
I20260924 16:12:27.304644 725 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.234.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab.
Entry for principal kudu/127.29.234.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab.
I20260924 16:12:27.364460 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.1:0
--local_ip_for_outbound_sockets=127.29.234.1
--webserver_interface=127.29.234.1
--webserver_port=0
--tserver_master_addrs=127.29.234.62:44055
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab
--principal=kudu/127.29.234.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:39223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:12:27.719293 737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:27.719568 737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:27.719659 737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:27.726733 737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:27.726845 737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:27.726956 737 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.1
I20260924 16:12:27.736780 737 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:39223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab
--principal=kudu/127.29.234.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.234.1
--webserver_port=0
--tserver_master_addrs=127.29.234.62:44055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.1
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:12:27.738117 737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:27.739390 737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:27.753144 743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:27.753222 746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:27.753165 742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:27.754024 737 server_base.cc:1034] running on GCE node
I20260924 16:12:27.754593 737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:27.755539 737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:27.756781 737 hybrid_clock.cc:648] HybridClock initialized: now 1790266347756712 us; error 57 us; skew 500 ppm
I20260924 16:12:27.757120 737 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:12:27 dist-test-slave-v5j9 krb5kdc[629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266347, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:27.760591 737 init.cc:377] Logged in from keytab as kudu/127.29.234.1@KRBTEST.COM (short username kudu)
I20260924 16:12:27.762221 737 webserver.cc:460] Webserver started at http://127.29.234.1:42377/ using document root <none> and password file <none>
I20260924 16:12:27.763003 737 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:27.763139 737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:27.763418 737 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:27.766477 737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-0/data/instance:
uuid: "d9c81e47cc58430a8b80f76dfef56a3c"
format_stamp: "Formatted at 2026-09-24 16:12:27 on dist-test-slave-v5j9"
server_key: "8cdb60c6fb0b4c1c0184d71324968f8e"
server_key_iv: "dbd7d6d8efe58d4bbe7d9a14eb84f8a8"
server_key_version: "encryptionkey@0"
I20260924 16:12:27.767189 737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-0/wal/instance:
uuid: "d9c81e47cc58430a8b80f76dfef56a3c"
format_stamp: "Formatted at 2026-09-24 16:12:27 on dist-test-slave-v5j9"
server_key: "8cdb60c6fb0b4c1c0184d71324968f8e"
server_key_iv: "dbd7d6d8efe58d4bbe7d9a14eb84f8a8"
server_key_version: "encryptionkey@0"
I20260924 16:12:27.772413 737 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 16:12:27.776373 753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:27.777819 737 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:12:27.778000 737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-0/wal
uuid: "d9c81e47cc58430a8b80f76dfef56a3c"
format_stamp: "Formatted at 2026-09-24 16:12:27 on dist-test-slave-v5j9"
server_key: "8cdb60c6fb0b4c1c0184d71324968f8e"
server_key_iv: "dbd7d6d8efe58d4bbe7d9a14eb84f8a8"
server_key_version: "encryptionkey@0"
I20260924 16:12:27.778206 737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:27.798874 737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:27.802935 737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:27.803231 737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:27.804255 737 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:12:27.805701 737 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:12:27.805832 737 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:27.805967 737 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:12:27.806037 737 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:27.842586 737 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.1:46297
I20260924 16:12:27.842605 866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.1:46297 every 8 connection(s)
I20260924 16:12:27.844403 737 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
I20260924 16:12:27.849306 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 737
I20260924 16:12:27.849609 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-0/wal/instance
I20260924 16:12:27.850472 30632 external_mini_cluster.cc:1428] Setting key a6f14aecd12166362baefd390ebca5a4
Sep 24 16:12:27 dist-test-slave-v5j9 krb5kdc[629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266347, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.1@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:12:27.868148 869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:27.847769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.1:44705 (local address 127.29.234.62:44055)
0924 16:12:27.848342 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:27.848350 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:27.851677 (+ 3327us) server_negotiation.cc:407] Connection header received
0924 16:12:27.851735 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:27.851739 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:27.851802 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:27.851928 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:27.853736 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:27.854488 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:27.856226 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:27.856655 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:27.856803 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:27.860733 (+ 3930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:27.860742 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:27.860750 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:27.860797 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:27.864558 (+ 3761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:27.865380 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:27.865389 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:27.865395 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:27.865548 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:27.865980 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:27.865988 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:27.865995 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:27.866435 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:27.866605 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:27.867345 (+ 740us) server_negotiation.cc:299] Negotiation successful
0924 16:12:27.867618 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":275,"threads_started":1}
I20260924 16:12:27.868726 870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:27.848572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:44055 (local address 127.29.234.1:44705)
0924 16:12:27.849280 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:27.849323 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:27.851394 (+ 2071us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:27.852189 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:27.852197 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:27.852903 (+ 706us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:27.853563 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:27.853584 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:27.854644 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:27.854649 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:27.855901 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:27.855913 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:27.856898 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:27.856908 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:27.857163 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:27.857919 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:27.857943 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:27.860464 (+ 2521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:27.864721 (+ 4257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:27.864729 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:27.864745 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:27.865202 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:27.865656 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:27.865659 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:27.865661 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:27.865848 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:27.866563 (+ 715us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:27.866570 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:27.866912 (+ 342us) client_negotiation.cc:769] Sending connection context
0924 16:12:27.867146 (+ 234us) client_negotiation.cc:240] Negotiation successful
0924 16:12:27.867568 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":742,"threads_started":1}
I20260924 16:12:27.870616 867 heartbeater.cc:347] Connected to a master server at 127.29.234.62:44055
WARNING: no policy specified for kudu/127.29.234.2@KRBTEST.COM; defaulting to no policy
I20260924 16:12:27.870995 867 heartbeater.cc:464] Registering TS with master...
I20260924 16:12:27.871812 867 heartbeater.cc:511] Master 127.29.234.62:44055 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.2@KRBTEST.COM" created.
I20260924 16:12:27.874191 675 ts_manager.cc:194] Registered new tserver with Master: d9c81e47cc58430a8b80f76dfef56a3c (127.29.234.1:46297)
I20260924 16:12:27.875680 675 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.234.1@KRBTEST.COM'} at 127.29.234.1:44705
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab.
Entry for principal kudu/127.29.234.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab.
I20260924 16:12:27.897046 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.2:0
--local_ip_for_outbound_sockets=127.29.234.2
--webserver_interface=127.29.234.2
--webserver_port=0
--tserver_master_addrs=127.29.234.62:44055
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab
--principal=kudu/127.29.234.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:39223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:12:28.229887 873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:28.230158 873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:28.230319 873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:28.237766 873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:28.237875 873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:28.238057 873 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.2
I20260924 16:12:28.247970 873 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:39223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab
--principal=kudu/127.29.234.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.234.2
--webserver_port=0
--tserver_master_addrs=127.29.234.62:44055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.2
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:12:28.249308 873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:28.250712 873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:28.263804 878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:28.263710 881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:28.263746 879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:28.264669 873 server_base.cc:1034] running on GCE node
I20260924 16:12:28.265267 873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:28.266258 873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:28.267450 873 hybrid_clock.cc:648] HybridClock initialized: now 1790266348267414 us; error 31 us; skew 500 ppm
I20260924 16:12:28.267794 873 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:12:28 dist-test-slave-v5j9 krb5kdc[629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266348, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:28.270987 873 init.cc:377] Logged in from keytab as kudu/127.29.234.2@KRBTEST.COM (short username kudu)
I20260924 16:12:28.272534 873 webserver.cc:460] Webserver started at http://127.29.234.2:38353/ using document root <none> and password file <none>
I20260924 16:12:28.273236 873 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:28.273401 873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:28.273746 873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:28.276625 873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-1/data/instance:
uuid: "a2c3a68edd2c4c1f97764e1b41beaa92"
format_stamp: "Formatted at 2026-09-24 16:12:28 on dist-test-slave-v5j9"
server_key: "1613389e24be58befcee20b61443b3e3"
server_key_iv: "977c6dda24af04b8fdc89993a7886bd1"
server_key_version: "encryptionkey@0"
I20260924 16:12:28.277493 873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-1/wal/instance:
uuid: "a2c3a68edd2c4c1f97764e1b41beaa92"
format_stamp: "Formatted at 2026-09-24 16:12:28 on dist-test-slave-v5j9"
server_key: "1613389e24be58befcee20b61443b3e3"
server_key_iv: "977c6dda24af04b8fdc89993a7886bd1"
server_key_version: "encryptionkey@0"
I20260924 16:12:28.282821 873 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 16:12:28.286388 888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:28.287662 873 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 16:12:28.287848 873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-1/wal
uuid: "a2c3a68edd2c4c1f97764e1b41beaa92"
format_stamp: "Formatted at 2026-09-24 16:12:28 on dist-test-slave-v5j9"
server_key: "1613389e24be58befcee20b61443b3e3"
server_key_iv: "977c6dda24af04b8fdc89993a7886bd1"
server_key_version: "encryptionkey@0"
I20260924 16:12:28.288043 873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:28.307313 873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:28.311252 873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:28.311535 873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:28.312489 873 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:12:28.313886 873 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:12:28.313987 873 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:28.314150 873 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:12:28.314219 873 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:28.345928 873 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.2:34657
I20260924 16:12:28.346009 1002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.2:34657 every 8 connection(s)
I20260924 16:12:28.347805 873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
I20260924 16:12:28.356577 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 873
I20260924 16:12:28.356782 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-1/wal/instance
I20260924 16:12:28.357328 30632 external_mini_cluster.cc:1428] Setting key 3c3912b40e947294d6c40a9c3e6999c9
Sep 24 16:12:28 dist-test-slave-v5j9 krb5kdc[629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266348, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.2@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:12:28.369846 869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:28.350640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.2:46191 (local address 127.29.234.62:44055)
0924 16:12:28.350871 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:28.350877 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:28.352114 (+ 1237us) server_negotiation.cc:407] Connection header received
0924 16:12:28.353949 (+ 1835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:28.353953 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:28.354025 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:28.354116 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:28.355618 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:28.356317 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:28.358479 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:28.359015 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:28.359158 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:28.363274 (+ 4116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:28.363280 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:28.363283 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:28.363318 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:28.366541 (+ 3223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:28.367352 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:28.367358 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:28.367360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:28.367488 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:28.367949 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:28.367958 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:28.367960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:28.368293 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:28.368452 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:28.369206 (+ 754us) server_negotiation.cc:299] Negotiation successful
0924 16:12:28.369468 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":82}
I20260924 16:12:28.371091 1005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:28.351240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:44055 (local address 127.29.234.2:46191)
0924 16:12:28.351944 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:28.351990 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:28.353682 (+ 1692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:28.354344 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:28.354352 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:28.354885 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:28.355456 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:28.355473 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:28.356476 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:28.356481 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:28.358326 (+ 1845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:28.358339 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:28.359253 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:28.359261 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:28.359496 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:28.360279 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:28.360306 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:28.362995 (+ 2689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:28.366699 (+ 3704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:28.366710 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:28.366730 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:28.367190 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:28.367609 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:28.367614 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:28.367617 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:28.367825 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:28.368431 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:28.368440 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:28.368830 (+ 390us) client_negotiation.cc:769] Sending connection context
0924 16:12:28.369113 (+ 283us) client_negotiation.cc:240] Negotiation successful
0924 16:12:28.369736 (+ 623us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":253,"threads_started":1}
I20260924 16:12:28.372800 1003 heartbeater.cc:347] Connected to a master server at 127.29.234.62:44055
I20260924 16:12:28.373158 1003 heartbeater.cc:464] Registering TS with master...
I20260924 16:12:28.374043 1003 heartbeater.cc:511] Master 127.29.234.62:44055 requested a full tablet report, sending...
I20260924 16:12:28.375674 675 ts_manager.cc:194] Registered new tserver with Master: a2c3a68edd2c4c1f97764e1b41beaa92 (127.29.234.2:34657)
I20260924 16:12:28.376513 675 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.234.2@KRBTEST.COM'} at 127.29.234.2:46191
WARNING: no policy specified for kudu/127.29.234.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab.
Entry for principal kudu/127.29.234.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab.
I20260924 16:12:28.402709 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.3:0
--local_ip_for_outbound_sockets=127.29.234.3
--webserver_interface=127.29.234.3
--webserver_port=0
--tserver_master_addrs=127.29.234.62:44055
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab
--principal=kudu/127.29.234.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:39223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:12:28.731787 1008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:28.732088 1008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:28.732192 1008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:28.739467 1008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:28.739580 1008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:28.739696 1008 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.3
I20260924 16:12:28.749390 1008 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:39223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab
--principal=kudu/127.29.234.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.234.3
--webserver_port=0
--tserver_master_addrs=127.29.234.62:44055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.3
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:12:28.750628 1008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:28.751979 1008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:28.765674 1017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:28.765705 1013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:28.765674 1015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:28.766613 1008 server_base.cc:1034] running on GCE node
I20260924 16:12:28.767238 1008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:28.768153 1008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:28.769418 1008 hybrid_clock.cc:648] HybridClock initialized: now 1790266348769292 us; error 55 us; skew 500 ppm
I20260924 16:12:28.769932 1008 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:12:28 dist-test-slave-v5j9 krb5kdc[629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266348, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:28.773058 1008 init.cc:377] Logged in from keytab as kudu/127.29.234.3@KRBTEST.COM (short username kudu)
I20260924 16:12:28.774509 1008 webserver.cc:460] Webserver started at http://127.29.234.3:40511/ using document root <none> and password file <none>
I20260924 16:12:28.775208 1008 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:28.775349 1008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:28.775664 1008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:28.778640 1008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-2/data/instance:
uuid: "0c92c0c8a2f14e8280111425ef99b939"
format_stamp: "Formatted at 2026-09-24 16:12:28 on dist-test-slave-v5j9"
server_key: "5814db6dfd7807ee5d95d49f99b21f9b"
server_key_iv: "9960e1dfc7392beb0d6958b414a57cbc"
server_key_version: "encryptionkey@0"
I20260924 16:12:28.779556 1008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-2/wal/instance:
uuid: "0c92c0c8a2f14e8280111425ef99b939"
format_stamp: "Formatted at 2026-09-24 16:12:28 on dist-test-slave-v5j9"
server_key: "5814db6dfd7807ee5d95d49f99b21f9b"
server_key_iv: "9960e1dfc7392beb0d6958b414a57cbc"
server_key_version: "encryptionkey@0"
I20260924 16:12:28.784997 1008 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 16:12:28.788508 1024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:28.789913 1008 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:12:28.790112 1008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-2/wal
uuid: "0c92c0c8a2f14e8280111425ef99b939"
format_stamp: "Formatted at 2026-09-24 16:12:28 on dist-test-slave-v5j9"
server_key: "5814db6dfd7807ee5d95d49f99b21f9b"
server_key_iv: "9960e1dfc7392beb0d6958b414a57cbc"
server_key_version: "encryptionkey@0"
I20260924 16:12:28.790338 1008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:28.801594 1008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:28.805402 1008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:28.805692 1008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:28.806629 1008 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:12:28.807909 1008 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:12:28.808010 1008 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:28.808163 1008 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:12:28.808235 1008 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:28.840696 1008 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.3:38743
I20260924 16:12:28.840734 1138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.3:38743 every 8 connection(s)
I20260924 16:12:28.842271 1008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
I20260924 16:12:28.851430 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 1008
I20260924 16:12:28.851640 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-2/wal/instance
I20260924 16:12:28.852066 30632 external_mini_cluster.cc:1428] Setting key 723ef147d7522dc477bffeb5b39835b1
Sep 24 16:12:28 dist-test-slave-v5j9 krb5kdc[629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266348, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.3@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:12:28.861896 869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:28.844921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.3:34023 (local address 127.29.234.62:44055)
0924 16:12:28.845124 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:28.845130 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:28.846571 (+ 1441us) server_negotiation.cc:407] Connection header received
0924 16:12:28.848514 (+ 1943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:28.848519 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:28.848599 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:28.848689 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:28.850209 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:28.850958 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:28.852582 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:28.852999 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:28.853151 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:28.856453 (+ 3302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:28.856458 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:28.856461 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:28.856491 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:28.858913 (+ 2422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:28.859610 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:28.859615 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:28.859617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:28.859747 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:28.860193 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:28.860205 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:28.860208 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:28.860481 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:28.860640 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:28.861304 (+ 664us) server_negotiation.cc:299] Negotiation successful
0924 16:12:28.861547 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 16:12:28.862867 1141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:28.845561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:44055 (local address 127.29.234.3:34023)
0924 16:12:28.846383 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:28.846437 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:28.848249 (+ 1812us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:28.848909 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:28.848917 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:28.849497 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:28.850063 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:28.850080 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:28.851083 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:28.851086 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:28.852429 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:28.852445 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:28.853236 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:28.853245 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:28.853521 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:28.854198 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:28.854220 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:28.856252 (+ 2032us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:28.859048 (+ 2796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:28.859057 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:28.859072 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:28.859477 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:28.859875 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:28.859880 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:28.859884 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:28.860061 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:28.860610 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:28.860618 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:28.860933 (+ 315us) client_negotiation.cc:769] Sending connection context
0924 16:12:28.861192 (+ 259us) client_negotiation.cc:240] Negotiation successful
0924 16:12:28.861724 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":308,"threads_started":1}
I20260924 16:12:28.864315 1139 heartbeater.cc:347] Connected to a master server at 127.29.234.62:44055
I20260924 16:12:28.864602 1139 heartbeater.cc:464] Registering TS with master...
I20260924 16:12:28.865238 1139 heartbeater.cc:511] Master 127.29.234.62:44055 requested a full tablet report, sending...
I20260924 16:12:28.866768 675 ts_manager.cc:194] Registered new tserver with Master: 0c92c0c8a2f14e8280111425ef99b939 (127.29.234.3:38743)
I20260924 16:12:28.867596 675 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.234.3@KRBTEST.COM'} at 127.29.234.3:34023
I20260924 16:12:28.879000 867 heartbeater.cc:503] Master 127.29.234.62:44055 was elected leader, sending a full tablet report...
I20260924 16:12:28.879243 30632 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 16:12:28.910249 869 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:12:28.907565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39940 (local address 127.29.234.62:44055)
0924 16:12:28.907805 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:28.907814 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:28.908244 (+ 430us) server_negotiation.cc:407] Connection header received
0924 16:12:28.908458 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:28.908463 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:28.908532 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:28.908640 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:28.909957 (+ 1317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39940: BlockingRecv error: recv got EOF from 127.0.0.1:39940 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 16:12:28.913159 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 737
W20260924 16:12:28.915863 30632 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 737: Not authorized: Client connection negotiation failed: client connection to 127.29.234.1:46297: server requires authentication, but client does not have Kerberos credentials available
I20260924 16:12:28.916064 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 737
I20260924 16:12:28.932579 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 873
W20260924 16:12:28.935400 30632 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 873: Not authorized: Client connection negotiation failed: client connection to 127.29.234.2:34657: server requires authentication, but client does not have Kerberos credentials available
I20260924 16:12:28.935570 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 873
I20260924 16:12:28.951510 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 1008
W20260924 16:12:28.954491 30632 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 1008: Not authorized: Client connection negotiation failed: client connection to 127.29.234.3:38743: server requires authentication, but client does not have Kerberos credentials available
I20260924 16:12:28.954625 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 1008
I20260924 16:12:28.970638 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 642
I20260924 16:12:28.971871 706 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:12:29.076637 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 642
2026-09-24T16:12:29Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3594 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790266323169505-30632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:12:29 dist-test-slave-v5j9 krb5kdc[1156](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:12:29 dist-test-slave-v5j9 krb5kdc[1156](info): set up 2 sockets
Sep 24 16:12:29 dist-test-slave-v5j9 krb5kdc[1156](info): commencing operation
krb5kdc: starting...
W20260924 16:12:30.176801 30632 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Sep 24 16:12:30 dist-test-slave-v5j9 krb5kdc[1156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:12:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:12:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.234.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab.
Entry for principal kudu/127.29.234.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab.
I20260924 16:12:30.385839 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790266323169505-30632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790266323169505-30632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790266323169505-30632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:45821
--webserver_interface=127.29.234.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab
--principal=kudu/127.29.234.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:38279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.234.62:45821
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:12:30.709729 1169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:30.710024 1169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:30.710112 1169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:30.717172 1169 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:12:30.717301 1169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:30.717366 1169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:30.717482 1169 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:12:30.717537 1169 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:12:30.727380 1169 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:38279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790266323169505-30632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790266323169505-30632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.234.62:45821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab
--principal=kudu/127.29.234.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:45821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.234.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790266323169505-30632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:12:30.728669 1169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:30.730001 1169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:30.742228 1174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:30.742285 1177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:30.742229 1175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:30.743043 1169 server_base.cc:1034] running on GCE node
I20260924 16:12:30.743850 1169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:30.745275 1169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:30.746531 1169 hybrid_clock.cc:648] HybridClock initialized: now 1790266350746474 us; error 44 us; skew 500 ppm
I20260924 16:12:30.746891 1169 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:12:30 dist-test-slave-v5j9 krb5kdc[1156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266350, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:30.749945 1169 init.cc:377] Logged in from keytab as kudu/127.29.234.62@KRBTEST.COM (short username kudu)
I20260924 16:12:30.751317 1169 webserver.cc:460] Webserver started at http://127.29.234.62:39221/ using document root <none> and password file <none>
I20260924 16:12:30.752007 1169 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:30.752151 1169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:30.752465 1169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:30.755445 1169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790266323169505-30632-0/minicluster-data/master-0/data/instance:
uuid: "995b9af5d0da404a9524127cf4bb611f"
format_stamp: "Formatted at 2026-09-24 16:12:30 on dist-test-slave-v5j9"
server_key: "16ce597a7043e6254fb0164c81ab690f"
server_key_iv: "20cb50a06494b868d9c6dfbe4d11b2f0"
server_key_version: "encryptionkey@0"
I20260924 16:12:30.756237 1169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790266323169505-30632-0/minicluster-data/master-0/wal/instance:
uuid: "995b9af5d0da404a9524127cf4bb611f"
format_stamp: "Formatted at 2026-09-24 16:12:30 on dist-test-slave-v5j9"
server_key: "16ce597a7043e6254fb0164c81ab690f"
server_key_iv: "20cb50a06494b868d9c6dfbe4d11b2f0"
server_key_version: "encryptionkey@0"
I20260924 16:12:30.761690 1169 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260924 16:12:30.765029 1185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:30.766436 1169 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:12:30.766619 1169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790266323169505-30632-0/minicluster-data/master-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790266323169505-30632-0/minicluster-data/master-0/wal
uuid: "995b9af5d0da404a9524127cf4bb611f"
format_stamp: "Formatted at 2026-09-24 16:12:30 on dist-test-slave-v5j9"
server_key: "16ce597a7043e6254fb0164c81ab690f"
server_key_iv: "20cb50a06494b868d9c6dfbe4d11b2f0"
server_key_version: "encryptionkey@0"
I20260924 16:12:30.766813 1169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790266323169505-30632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790266323169505-30632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790266323169505-30632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:30.785127 1169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:30.789551 1169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:30.789844 1169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:30.808286 1169 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.62:45821
I20260924 16:12:30.808310 1237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.62:45821 every 8 connection(s)
I20260924 16:12:30.809716 1169 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
I20260924 16:12:30.813473 1238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:30.814180 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 1169
I20260924 16:12:30.814450 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790266323169505-30632-0/minicluster-data/master-0/wal/instance
I20260924 16:12:30.815311 30632 external_mini_cluster.cc:1428] Setting key 3ce473505a69cc0f659a3c66ab814325
I20260924 16:12:30.825984 1238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 995b9af5d0da404a9524127cf4bb611f: Bootstrap starting.
Sep 24 16:12:30 dist-test-slave-v5j9 krb5kdc[1156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:12:30.829133 1238 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 995b9af5d0da404a9524127cf4bb611f: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:30.830219 1238 log.cc:824] T 00000000000000000000000000000000 P 995b9af5d0da404a9524127cf4bb611f: Log is configured to *not* fsync() on all Append() calls
I20260924 16:12:30.833065 1238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 995b9af5d0da404a9524127cf4bb611f: No bootstrap required, opened a new log
I20260924 16:12:30.837177 1241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:30.817236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56818 (local address 127.29.234.62:45821)
0924 16:12:30.818095 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:30.818107 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:30.818145 (+ 38us) server_negotiation.cc:407] Connection header received
0924 16:12:30.820021 (+ 1876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:30.820043 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:30.820433 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:30.820800 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:30.822264 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:30.823579 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:30.824827 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:30.825571 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:30.825770 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:30.829849 (+ 4079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:30.829863 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:30.829878 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:30.829919 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:30.832914 (+ 2995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:30.833703 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:30.833719 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:30.833725 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:30.833885 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:30.834266 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:30.834270 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:30.834272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:30.834814 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:30.835169 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:30.835670 (+ 501us) server_negotiation.cc:299] Negotiation successful
0924 16:12:30.836085 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":330,"threads_started":1}
I20260924 16:12:30.841122 30632 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:12:30.841254 1238 raft_consensus.cc:348] T 00000000000000000000000000000000 P 995b9af5d0da404a9524127cf4bb611f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "995b9af5d0da404a9524127cf4bb611f" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 45821 } }
I20260924 16:12:30.841697 1238 raft_consensus.cc:374] T 00000000000000000000000000000000 P 995b9af5d0da404a9524127cf4bb611f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:30.841806 1238 raft_consensus.cc:729] T 00000000000000000000000000000000 P 995b9af5d0da404a9524127cf4bb611f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 995b9af5d0da404a9524127cf4bb611f, State: Initialized, Role: FOLLOWER
I20260924 16:12:30.842350 1238 consensus_queue.cc:260] T 00000000000000000000000000000000 P 995b9af5d0da404a9524127cf4bb611f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "995b9af5d0da404a9524127cf4bb611f" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 45821 } }
I20260924 16:12:30.842585 1238 raft_consensus.cc:388] T 00000000000000000000000000000000 P 995b9af5d0da404a9524127cf4bb611f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:12:30.842722 1238 raft_consensus.cc:482] T 00000000000000000000000000000000 P 995b9af5d0da404a9524127cf4bb611f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:12:30.842904 1238 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 995b9af5d0da404a9524127cf4bb611f [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:30.844985 1238 raft_consensus.cc:504] T 00000000000000000000000000000000 P 995b9af5d0da404a9524127cf4bb611f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "995b9af5d0da404a9524127cf4bb611f" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 45821 } }
I20260924 16:12:30.845566 1238 leader_election.cc:302] T 00000000000000000000000000000000 P 995b9af5d0da404a9524127cf4bb611f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 995b9af5d0da404a9524127cf4bb611f; no voters:
I20260924 16:12:30.846266 1238 leader_election.cc:288] T 00000000000000000000000000000000 P 995b9af5d0da404a9524127cf4bb611f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:12:30.846388 1243 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 995b9af5d0da404a9524127cf4bb611f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:12:30.847966 1243 raft_consensus.cc:686] T 00000000000000000000000000000000 P 995b9af5d0da404a9524127cf4bb611f [term 1 LEADER]: Becoming Leader. State: Replica: 995b9af5d0da404a9524127cf4bb611f, State: Running, Role: LEADER
I20260924 16:12:30.848512 1243 consensus_queue.cc:237] T 00000000000000000000000000000000 P 995b9af5d0da404a9524127cf4bb611f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "995b9af5d0da404a9524127cf4bb611f" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 45821 } }
I20260924 16:12:30.848821 1238 sys_catalog.cc:564] T 00000000000000000000000000000000 P 995b9af5d0da404a9524127cf4bb611f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:12:30.854607 1244 sys_catalog.cc:455] T 00000000000000000000000000000000 P 995b9af5d0da404a9524127cf4bb611f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "995b9af5d0da404a9524127cf4bb611f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "995b9af5d0da404a9524127cf4bb611f" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 45821 } } }
I20260924 16:12:30.855039 1244 sys_catalog.cc:458] T 00000000000000000000000000000000 P 995b9af5d0da404a9524127cf4bb611f [sys.catalog]: This master's current role is: LEADER
I20260924 16:12:30.867013 1253 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:12:30.866801 1245 sys_catalog.cc:455] T 00000000000000000000000000000000 P 995b9af5d0da404a9524127cf4bb611f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 995b9af5d0da404a9524127cf4bb611f. Latest consensus state: current_term: 1 leader_uuid: "995b9af5d0da404a9524127cf4bb611f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "995b9af5d0da404a9524127cf4bb611f" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 45821 } } }
I20260924 16:12:30.867106 1245 sys_catalog.cc:458] T 00000000000000000000000000000000 P 995b9af5d0da404a9524127cf4bb611f [sys.catalog]: This master's current role is: LEADER
I20260924 16:12:30.871981 1253 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:12:30.897440 1253 catalog_manager.cc:1347] Generated new cluster ID: 0776069433c44ce8a548634eb6c6234b
I20260924 16:12:30.897611 1253 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:12:30.903302 1266 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 16:12:30.912449 1253 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:12:30.914109 1253 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:12:30.921525 1241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:30.912026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56826 (local address 127.29.234.62:45821)
0924 16:12:30.912200 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:30.912207 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:30.912804 (+ 597us) server_negotiation.cc:407] Connection header received
0924 16:12:30.913009 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:30.913014 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:30.913086 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:30.913211 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:30.914524 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:30.915620 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:30.917517 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:30.918190 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:30.918308 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:30.919578 (+ 1270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:30.919584 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:30.919589 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:12:30.920460 (+ 871us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:30.920543 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:12:30.920570 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:30.920669 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:30.920841 (+ 172us) server_negotiation.cc:299] Negotiation successful
0924 16:12:30.920966 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 16:12:30.928659 1253 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 995b9af5d0da404a9524127cf4bb611f: Generated new TSK 0
I20260924 16:12:30.929674 1253 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:12:30.941994 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 1169
I20260924 16:12:30.943383 1233 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:12:31.035276 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 1169
2026-09-24T16:12:31Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1953 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:12:31 dist-test-slave-v5j9 krb5kdc[1273](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:12:31 dist-test-slave-v5j9 krb5kdc[1273](info): set up 2 sockets
Sep 24 16:12:31 dist-test-slave-v5j9 krb5kdc[1273](info): commencing operation
krb5kdc: starting...
W20260924 16:12:32.137120 30632 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Sep 24 16:12:32 dist-test-slave-v5j9 krb5kdc[1273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:12:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:12:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.234.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab.
Entry for principal kudu/127.29.234.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab.
I20260924 16:12:32.329715 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:42873
--webserver_interface=127.29.234.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab
--principal=kudu/127.29.234.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:44317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.234.62:42873
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:12:32.653040 1286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:32.653358 1286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:32.653517 1286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:32.661398 1286 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:12:32.661507 1286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:32.661619 1286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:32.661727 1286 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:12:32.661792 1286 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:12:32.671945 1286 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:44317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.234.62:42873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab
--principal=kudu/127.29.234.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:42873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.234.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:12:32.673444 1286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:32.674844 1286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:32.687179 1291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:32.687240 1292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:32.687361 1294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:32.688045 1286 server_base.cc:1034] running on GCE node
I20260924 16:12:32.688841 1286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:32.690335 1286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:32.691573 1286 hybrid_clock.cc:648] HybridClock initialized: now 1790266352691519 us; error 60 us; skew 500 ppm
I20260924 16:12:32.691936 1286 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:12:32 dist-test-slave-v5j9 krb5kdc[1273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266352, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:32.695071 1286 init.cc:377] Logged in from keytab as kudu/127.29.234.62@KRBTEST.COM (short username kudu)
I20260924 16:12:32.696523 1286 webserver.cc:460] Webserver started at http://127.29.234.62:43247/ using document root <none> and password file <none>
I20260924 16:12:32.697204 1286 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:32.697408 1286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:32.697747 1286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:32.700657 1286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/minicluster-data/master-0/data/instance:
uuid: "8b9768eaf3064f6d8d690632aa87c567"
format_stamp: "Formatted at 2026-09-24 16:12:32 on dist-test-slave-v5j9"
server_key: "1c37bcb22528d360909d1031078dd260"
server_key_iv: "c6b9547fc01c9d1c5377fab1d8ce6ae5"
server_key_version: "encryptionkey@0"
I20260924 16:12:32.701494 1286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/minicluster-data/master-0/wal/instance:
uuid: "8b9768eaf3064f6d8d690632aa87c567"
format_stamp: "Formatted at 2026-09-24 16:12:32 on dist-test-slave-v5j9"
server_key: "1c37bcb22528d360909d1031078dd260"
server_key_iv: "c6b9547fc01c9d1c5377fab1d8ce6ae5"
server_key_version: "encryptionkey@0"
I20260924 16:12:32.706867 1286 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260924 16:12:32.710326 1301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:32.711650 1286 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 16:12:32.711855 1286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/minicluster-data/master-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/minicluster-data/master-0/wal
uuid: "8b9768eaf3064f6d8d690632aa87c567"
format_stamp: "Formatted at 2026-09-24 16:12:32 on dist-test-slave-v5j9"
server_key: "1c37bcb22528d360909d1031078dd260"
server_key_iv: "c6b9547fc01c9d1c5377fab1d8ce6ae5"
server_key_version: "encryptionkey@0"
I20260924 16:12:32.712066 1286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:32.747989 1286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:32.752624 1286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:32.752934 1286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:32.772223 1286 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.62:42873
I20260924 16:12:32.772322 1354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.62:42873 every 8 connection(s)
I20260924 16:12:32.773814 1286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
I20260924 16:12:32.777725 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:32.778241 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 1286
I20260924 16:12:32.778487 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/minicluster-data/master-0/wal/instance
I20260924 16:12:32.779143 30632 external_mini_cluster.cc:1428] Setting key 361d96980f02f94abab73a1b2da7f84a
I20260924 16:12:32.792086 1355 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b9768eaf3064f6d8d690632aa87c567: Bootstrap starting.
Sep 24 16:12:32 dist-test-slave-v5j9 krb5kdc[1273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:12:32.795282 1355 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b9768eaf3064f6d8d690632aa87c567: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:32.796655 1355 log.cc:824] T 00000000000000000000000000000000 P 8b9768eaf3064f6d8d690632aa87c567: Log is configured to *not* fsync() on all Append() calls
I20260924 16:12:32.800148 1355 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b9768eaf3064f6d8d690632aa87c567: No bootstrap required, opened a new log
I20260924 16:12:32.803898 1358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:32.781277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40150 (local address 127.29.234.62:42873)
0924 16:12:32.782136 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:32.782151 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:32.782204 (+ 53us) server_negotiation.cc:407] Connection header received
0924 16:12:32.784420 (+ 2216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:32.784452 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:32.784975 (+ 523us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:32.785471 (+ 496us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:32.786783 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:32.788812 (+ 2029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:32.790085 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:32.791089 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:32.791259 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:32.795207 (+ 3948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:32.795218 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:32.795239 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:32.795293 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:32.799112 (+ 3819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:32.799763 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:32.799772 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:32.799780 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:32.799984 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:32.800412 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:32.800417 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:32.800420 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:32.801166 (+ 746us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:32.801527 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:32.801879 (+ 352us) server_negotiation.cc:299] Negotiation successful
0924 16:12:32.802364 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":325,"threads_started":1}
I20260924 16:12:32.809569 1355 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b9768eaf3064f6d8d690632aa87c567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b9768eaf3064f6d8d690632aa87c567" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 42873 } }
I20260924 16:12:32.809898 1355 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b9768eaf3064f6d8d690632aa87c567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:32.809970 1355 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b9768eaf3064f6d8d690632aa87c567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b9768eaf3064f6d8d690632aa87c567, State: Initialized, Role: FOLLOWER
I20260924 16:12:32.810534 1355 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b9768eaf3064f6d8d690632aa87c567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b9768eaf3064f6d8d690632aa87c567" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 42873 } }
I20260924 16:12:32.810730 1355 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b9768eaf3064f6d8d690632aa87c567 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:12:32.810825 1355 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b9768eaf3064f6d8d690632aa87c567 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:12:32.810953 1355 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b9768eaf3064f6d8d690632aa87c567 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:32.813233 1355 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b9768eaf3064f6d8d690632aa87c567 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b9768eaf3064f6d8d690632aa87c567" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 42873 } }
I20260924 16:12:32.813846 1355 leader_election.cc:302] T 00000000000000000000000000000000 P 8b9768eaf3064f6d8d690632aa87c567 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b9768eaf3064f6d8d690632aa87c567; no voters:
I20260924 16:12:32.814678 1355 leader_election.cc:288] T 00000000000000000000000000000000 P 8b9768eaf3064f6d8d690632aa87c567 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:12:32.814801 1360 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b9768eaf3064f6d8d690632aa87c567 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:12:32.815503 1360 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b9768eaf3064f6d8d690632aa87c567 [term 1 LEADER]: Becoming Leader. State: Replica: 8b9768eaf3064f6d8d690632aa87c567, State: Running, Role: LEADER
I20260924 16:12:32.816150 1360 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b9768eaf3064f6d8d690632aa87c567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b9768eaf3064f6d8d690632aa87c567" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 42873 } }
I20260924 16:12:32.817039 1355 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b9768eaf3064f6d8d690632aa87c567 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:12:32.819073 1362 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b9768eaf3064f6d8d690632aa87c567 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b9768eaf3064f6d8d690632aa87c567. Latest consensus state: current_term: 1 leader_uuid: "8b9768eaf3064f6d8d690632aa87c567" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b9768eaf3064f6d8d690632aa87c567" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 42873 } } }
I20260924 16:12:32.819473 1362 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b9768eaf3064f6d8d690632aa87c567 [sys.catalog]: This master's current role is: LEADER
I20260924 16:12:32.821574 1361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b9768eaf3064f6d8d690632aa87c567 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b9768eaf3064f6d8d690632aa87c567" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b9768eaf3064f6d8d690632aa87c567" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 42873 } } }
I20260924 16:12:32.822013 1361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b9768eaf3064f6d8d690632aa87c567 [sys.catalog]: This master's current role is: LEADER
I20260924 16:12:32.826732 1369 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:12:32.831866 1369 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:12:32.840394 1369 catalog_manager.cc:1347] Generated new cluster ID: d6dad50fd0104f4694f36325d72fc867
I20260924 16:12:32.840531 1369 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:12:32.850994 1369 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:12:32.852829 1369 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:12:32.863601 1369 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b9768eaf3064f6d8d690632aa87c567: Generated new TSK 0
I20260924 16:12:32.864414 1369 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.234.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab.
Entry for principal kudu/127.29.234.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab.
I20260924 16:12:32.921126 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.1:0
--local_ip_for_outbound_sockets=127.29.234.1
--webserver_interface=127.29.234.1
--webserver_port=0
--tserver_master_addrs=127.29.234.62:42873
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab
--principal=kudu/127.29.234.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:44317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:12:33.246083 1382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:33.246379 1382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:33.246472 1382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:33.253737 1382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:33.253835 1382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:33.253943 1382 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.1
I20260924 16:12:33.263412 1382 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:44317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab
--principal=kudu/127.29.234.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.234.1
--webserver_port=0
--tserver_master_addrs=127.29.234.62:42873
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.1
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:12:33.264599 1382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:33.265913 1382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:33.279110 1388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:33.279115 1390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:33.279152 1387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:33.280005 1382 server_base.cc:1034] running on GCE node
I20260924 16:12:33.280646 1382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:33.281705 1382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:33.282914 1382 hybrid_clock.cc:648] HybridClock initialized: now 1790266353282857 us; error 64 us; skew 500 ppm
I20260924 16:12:33.283252 1382 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:12:33 dist-test-slave-v5j9 krb5kdc[1273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266353, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:33.286650 1382 init.cc:377] Logged in from keytab as kudu/127.29.234.1@KRBTEST.COM (short username kudu)
I20260924 16:12:33.288110 1382 webserver.cc:460] Webserver started at http://127.29.234.1:41859/ using document root <none> and password file <none>
I20260924 16:12:33.288811 1382 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:33.288959 1382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:33.289274 1382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:33.292201 1382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-0/data/instance:
uuid: "215513e4b72d47ffa1a85fc4de3fe8a8"
format_stamp: "Formatted at 2026-09-24 16:12:33 on dist-test-slave-v5j9"
server_key: "ea7143b4afe1de21b09a6eb305cacecc"
server_key_iv: "c01ea436db72dd088a62ff413c094f89"
server_key_version: "encryptionkey@0"
I20260924 16:12:33.292995 1382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-0/wal/instance:
uuid: "215513e4b72d47ffa1a85fc4de3fe8a8"
format_stamp: "Formatted at 2026-09-24 16:12:33 on dist-test-slave-v5j9"
server_key: "ea7143b4afe1de21b09a6eb305cacecc"
server_key_iv: "c01ea436db72dd088a62ff413c094f89"
server_key_version: "encryptionkey@0"
I20260924 16:12:33.298257 1382 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 16:12:33.301671 1397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:33.303017 1382 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 16:12:33.303192 1382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-0/wal
uuid: "215513e4b72d47ffa1a85fc4de3fe8a8"
format_stamp: "Formatted at 2026-09-24 16:12:33 on dist-test-slave-v5j9"
server_key: "ea7143b4afe1de21b09a6eb305cacecc"
server_key_iv: "c01ea436db72dd088a62ff413c094f89"
server_key_version: "encryptionkey@0"
I20260924 16:12:33.303390 1382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:33.316907 1382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:33.321254 1382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:33.321561 1382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:33.322469 1382 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:12:33.323843 1382 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:12:33.323915 1382 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:33.324081 1382 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:12:33.324167 1382 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:33.354928 1382 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.1:38885
I20260924 16:12:33.354964 1511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.1:38885 every 8 connection(s)
I20260924 16:12:33.356272 1382 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
I20260924 16:12:33.359547 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 1382
I20260924 16:12:33.359759 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-0/wal/instance
I20260924 16:12:33.360394 30632 external_mini_cluster.cc:1428] Setting key c05b699e85cbf40b9ab044992fe0e4e6
Sep 24 16:12:33 dist-test-slave-v5j9 krb5kdc[1273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266353, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.1@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:12:33.379782 1514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:33.358916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.1:50249 (local address 127.29.234.62:42873)
0924 16:12:33.359379 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:33.359385 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:33.360624 (+ 1239us) server_negotiation.cc:407] Connection header received
0924 16:12:33.362868 (+ 2244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:33.362874 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:33.362987 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:33.363097 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:33.365420 (+ 2323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:33.366219 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:33.368820 (+ 2601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:33.369594 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:33.369740 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:33.374000 (+ 4260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:33.374005 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:33.374009 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:33.374040 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:33.376583 (+ 2543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:33.377404 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:33.377412 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:33.377418 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:33.377547 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:33.377986 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:33.377992 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:33.377996 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:33.378277 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:33.378427 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:33.379162 (+ 735us) server_negotiation.cc:299] Negotiation successful
0924 16:12:33.379416 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":217,"threads_started":1}
WARNING: no policy specified for kudu/127.29.234.2@KRBTEST.COM; defaulting to no policy
I20260924 16:12:33.380812 1515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:33.359577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:42873 (local address 127.29.234.1:50249)
0924 16:12:33.360438 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:33.360493 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:33.362503 (+ 2010us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:33.363399 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:33.363414 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:33.364310 (+ 896us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:33.365187 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:33.365215 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:33.366389 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:33.366397 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:33.368601 (+ 2204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:33.368620 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:33.369832 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:33.369842 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:33.370189 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:33.370944 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:33.370978 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:33.373780 (+ 2802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:33.376734 (+ 2954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:33.376744 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:33.376770 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:33.377266 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:33.377662 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:33.377667 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:33.377671 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:33.377870 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:33.378415 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:33.378423 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:33.378789 (+ 366us) client_negotiation.cc:769] Sending connection context
0924 16:12:33.379062 (+ 273us) client_negotiation.cc:240] Negotiation successful
0924 16:12:33.379620 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":357,"threads_started":1}
I20260924 16:12:33.382416 1512 heartbeater.cc:347] Connected to a master server at 127.29.234.62:42873
I20260924 16:12:33.382761 1512 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.2@KRBTEST.COM" created.
I20260924 16:12:33.383688 1512 heartbeater.cc:511] Master 127.29.234.62:42873 requested a full tablet report, sending...
I20260924 16:12:33.386030 1318 ts_manager.cc:194] Registered new tserver with Master: 215513e4b72d47ffa1a85fc4de3fe8a8 (127.29.234.1:38885)
I20260924 16:12:33.387514 1318 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.234.1@KRBTEST.COM'} at 127.29.234.1:50249
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab.
Entry for principal kudu/127.29.234.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab.
I20260924 16:12:33.408023 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.2:0
--local_ip_for_outbound_sockets=127.29.234.2
--webserver_interface=127.29.234.2
--webserver_port=0
--tserver_master_addrs=127.29.234.62:42873
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab
--principal=kudu/127.29.234.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:44317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:12:33.729141 1518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:33.729535 1518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:33.729712 1518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:33.738422 1518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:33.738579 1518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:33.738811 1518 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.2
I20260924 16:12:33.752256 1518 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:44317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab
--principal=kudu/127.29.234.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.234.2
--webserver_port=0
--tserver_master_addrs=127.29.234.62:42873
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.2
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:12:33.753693 1518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:33.755028 1518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:33.770536 1523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:33.770566 1524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:33.771093 1527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:33.771772 1518 server_base.cc:1034] running on GCE node
I20260924 16:12:33.772564 1518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:33.773864 1518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:33.775146 1518 hybrid_clock.cc:648] HybridClock initialized: now 1790266353775082 us; error 45 us; skew 500 ppm
I20260924 16:12:33.775547 1518 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:12:33 dist-test-slave-v5j9 krb5kdc[1273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266353, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:33.779640 1518 init.cc:377] Logged in from keytab as kudu/127.29.234.2@KRBTEST.COM (short username kudu)
I20260924 16:12:33.781399 1518 webserver.cc:460] Webserver started at http://127.29.234.2:33295/ using document root <none> and password file <none>
I20260924 16:12:33.782328 1518 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:33.782492 1518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:33.782835 1518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:33.786880 1518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-1/data/instance:
uuid: "f02262c613034f27b9e5ea4a8779669a"
format_stamp: "Formatted at 2026-09-24 16:12:33 on dist-test-slave-v5j9"
server_key: "5bd24e16fa095c4f7a567d91f4d921d2"
server_key_iv: "a373ed96dd95ec454da377b3bbecd005"
server_key_version: "encryptionkey@0"
I20260924 16:12:33.787914 1518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-1/wal/instance:
uuid: "f02262c613034f27b9e5ea4a8779669a"
format_stamp: "Formatted at 2026-09-24 16:12:33 on dist-test-slave-v5j9"
server_key: "5bd24e16fa095c4f7a567d91f4d921d2"
server_key_iv: "a373ed96dd95ec454da377b3bbecd005"
server_key_version: "encryptionkey@0"
I20260924 16:12:33.794593 1518 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260924 16:12:33.798291 1534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:33.799765 1518 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:12:33.799937 1518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-1/wal
uuid: "f02262c613034f27b9e5ea4a8779669a"
format_stamp: "Formatted at 2026-09-24 16:12:33 on dist-test-slave-v5j9"
server_key: "5bd24e16fa095c4f7a567d91f4d921d2"
server_key_iv: "a373ed96dd95ec454da377b3bbecd005"
server_key_version: "encryptionkey@0"
I20260924 16:12:33.800154 1518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:33.832834 1518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:33.837625 1518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:33.837953 1518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:33.839042 1518 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:12:33.840428 1518 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:12:33.840533 1518 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:33.840674 1518 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:12:33.840763 1518 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:33.875882 1518 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.2:36255
I20260924 16:12:33.875937 1648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.2:36255 every 8 connection(s)
I20260924 16:12:33.877183 1518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
I20260924 16:12:33.879082 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 1518
I20260924 16:12:33.879271 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-1/wal/instance
I20260924 16:12:33.879733 30632 external_mini_cluster.cc:1428] Setting key 71f8643cd0237665507c57bbdef30bf8
Sep 24 16:12:33 dist-test-slave-v5j9 krb5kdc[1273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266353, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.2@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.234.3@KRBTEST.COM; defaulting to no policy
I20260924 16:12:33.899199 1514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:33.880003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.2:53993 (local address 127.29.234.62:42873)
0924 16:12:33.880228 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:33.880235 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:33.881653 (+ 1418us) server_negotiation.cc:407] Connection header received
0924 16:12:33.883568 (+ 1915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:33.883573 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:33.883660 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:33.883781 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:33.885288 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:33.886046 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:33.887632 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:33.888377 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:33.888519 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:33.891973 (+ 3454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:33.891981 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:33.891989 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:33.892034 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:33.895752 (+ 3718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:33.896533 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:33.896540 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:33.896545 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:33.896709 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:33.897125 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:33.897131 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:33.897136 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:33.897565 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:33.897732 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:33.898437 (+ 705us) server_negotiation.cc:299] Negotiation successful
0924 16:12:33.898673 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 16:12:33.899915 1651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:33.880638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:42873 (local address 127.29.234.2:53993)
0924 16:12:33.881505 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:33.881549 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:33.883304 (+ 1755us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:33.884027 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:33.884035 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:33.884539 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:33.885118 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:33.885135 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:33.886217 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:33.886224 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:33.887475 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:33.887486 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:33.888598 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:33.888603 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:33.888849 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:33.889620 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:33.889642 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:33.891735 (+ 2093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:33.895986 (+ 4251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:33.895992 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:33.896005 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:33.896406 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:33.896825 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:33.896828 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:33.896830 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:33.897008 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:33.897688 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:33.897693 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:33.898063 (+ 370us) client_negotiation.cc:769] Sending connection context
0924 16:12:33.898290 (+ 227us) client_negotiation.cc:240] Negotiation successful
0924 16:12:33.898672 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":347,"threads_started":1}
I20260924 16:12:33.901376 1649 heartbeater.cc:347] Connected to a master server at 127.29.234.62:42873
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.3@KRBTEST.COM" created.
I20260924 16:12:33.901700 1649 heartbeater.cc:464] Registering TS with master...
I20260924 16:12:33.902490 1649 heartbeater.cc:511] Master 127.29.234.62:42873 requested a full tablet report, sending...
I20260924 16:12:33.904088 1318 ts_manager.cc:194] Registered new tserver with Master: f02262c613034f27b9e5ea4a8779669a (127.29.234.2:36255)
I20260924 16:12:33.905018 1318 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.234.2@KRBTEST.COM'} at 127.29.234.2:53993
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab.
Entry for principal kudu/127.29.234.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab.
I20260924 16:12:33.925117 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.3:0
--local_ip_for_outbound_sockets=127.29.234.3
--webserver_interface=127.29.234.3
--webserver_port=0
--tserver_master_addrs=127.29.234.62:42873
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab
--principal=kudu/127.29.234.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:44317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:12:34.250345 1654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:34.250674 1654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:34.250873 1654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:34.258100 1654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:34.258248 1654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:34.258435 1654 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.3
I20260924 16:12:34.268018 1654 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:44317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab
--principal=kudu/127.29.234.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.234.3
--webserver_port=0
--tserver_master_addrs=127.29.234.62:42873
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.3
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:12:34.269387 1654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:34.270784 1654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:34.283723 1660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:34.283751 1659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:34.284199 1662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:34.284649 1654 server_base.cc:1034] running on GCE node
I20260924 16:12:34.285324 1654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:34.286307 1654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:34.287559 1654 hybrid_clock.cc:648] HybridClock initialized: now 1790266354287513 us; error 35 us; skew 500 ppm
I20260924 16:12:34.287909 1654 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:12:34 dist-test-slave-v5j9 krb5kdc[1273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266354, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:34.291306 1654 init.cc:377] Logged in from keytab as kudu/127.29.234.3@KRBTEST.COM (short username kudu)
I20260924 16:12:34.292768 1654 webserver.cc:460] Webserver started at http://127.29.234.3:35417/ using document root <none> and password file <none>
I20260924 16:12:34.293614 1654 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:34.293780 1654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:34.294078 1654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:34.297029 1654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-2/data/instance:
uuid: "05cc100be4814fa7b47e9bd7e7459048"
format_stamp: "Formatted at 2026-09-24 16:12:34 on dist-test-slave-v5j9"
server_key: "a75a013c3e4a480d228d65bb6a08bb06"
server_key_iv: "ac22800d751ff00dec96e485ae5d8aa3"
server_key_version: "encryptionkey@0"
I20260924 16:12:34.297911 1654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-2/wal/instance:
uuid: "05cc100be4814fa7b47e9bd7e7459048"
format_stamp: "Formatted at 2026-09-24 16:12:34 on dist-test-slave-v5j9"
server_key: "a75a013c3e4a480d228d65bb6a08bb06"
server_key_iv: "ac22800d751ff00dec96e485ae5d8aa3"
server_key_version: "encryptionkey@0"
I20260924 16:12:34.302945 1654 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 16:12:34.306831 1670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:34.308466 1654 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:12:34.308643 1654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-2/wal
uuid: "05cc100be4814fa7b47e9bd7e7459048"
format_stamp: "Formatted at 2026-09-24 16:12:34 on dist-test-slave-v5j9"
server_key: "a75a013c3e4a480d228d65bb6a08bb06"
server_key_iv: "ac22800d751ff00dec96e485ae5d8aa3"
server_key_version: "encryptionkey@0"
I20260924 16:12:34.308833 1654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:34.332589 1654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:34.337296 1654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:34.337643 1654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:34.338698 1654 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:12:34.340130 1654 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:12:34.340216 1654 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:34.340387 1654 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:12:34.340458 1654 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:34.375476 1654 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.3:44799
I20260924 16:12:34.375517 1783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.3:44799 every 8 connection(s)
I20260924 16:12:34.376811 1654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
I20260924 16:12:34.384492 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 1654
I20260924 16:12:34.384701 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790266323169505-30632-0/minicluster-data/ts-2/wal/instance
I20260924 16:12:34.385170 30632 external_mini_cluster.cc:1428] Setting key 8d702b161460622708a74f914022912c
I20260924 16:12:34.390489 1512 heartbeater.cc:503] Master 127.29.234.62:42873 was elected leader, sending a full tablet report...
Sep 24 16:12:34 dist-test-slave-v5j9 krb5kdc[1273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266354, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.3@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:12:34.397679 1514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:34.379433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.3:57471 (local address 127.29.234.62:42873)
0924 16:12:34.379645 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:34.379651 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:34.380910 (+ 1259us) server_negotiation.cc:407] Connection header received
0924 16:12:34.382794 (+ 1884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:34.382799 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:34.382894 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:34.382985 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:34.384824 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:34.385712 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:34.387597 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:34.388221 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:34.388317 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:34.391954 (+ 3637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:34.391959 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:34.391962 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:34.391994 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:34.394600 (+ 2606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:34.395328 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:34.395335 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:34.395339 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:34.395476 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:34.395904 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:34.395913 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:34.395916 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:34.396191 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:34.396352 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:34.397068 (+ 716us) server_negotiation.cc:299] Negotiation successful
0924 16:12:34.397289 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 16:12:34.398742 1786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:34.380026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:42873 (local address 127.29.234.3:57471)
0924 16:12:34.380747 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:34.380789 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:34.382545 (+ 1756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:34.383191 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:34.383199 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:34.383716 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:34.384572 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:34.384608 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:34.385880 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:34.385885 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:34.387441 (+ 1556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:34.387454 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:34.388444 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:34.388450 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:34.388650 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:34.389404 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:34.389437 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:34.391720 (+ 2283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:34.394742 (+ 3022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:34.394751 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:34.394768 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:34.395202 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:34.395589 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:34.395594 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:34.395598 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:34.395788 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:34.396322 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:34.396330 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:34.396690 (+ 360us) client_negotiation.cc:769] Sending connection context
0924 16:12:34.396975 (+ 285us) client_negotiation.cc:240] Negotiation successful
0924 16:12:34.397537 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":278,"threads_started":1}
I20260924 16:12:34.400286 1784 heartbeater.cc:347] Connected to a master server at 127.29.234.62:42873
I20260924 16:12:34.400655 1784 heartbeater.cc:464] Registering TS with master...
I20260924 16:12:34.401580 1784 heartbeater.cc:511] Master 127.29.234.62:42873 requested a full tablet report, sending...
I20260924 16:12:34.402904 1317 ts_manager.cc:194] Registered new tserver with Master: 05cc100be4814fa7b47e9bd7e7459048 (127.29.234.3:44799)
I20260924 16:12:34.403839 1317 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.234.3@KRBTEST.COM'} at 127.29.234.3:57471
I20260924 16:12:34.413506 30632 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 16:12:34 dist-test-slave-v5j9 krb5kdc[1273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266354, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 24 16:12:34 dist-test-slave-v5j9 krb5kdc[1273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266354, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:12:34.461876 1514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:34.448330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40162 (local address 127.29.234.62:42873)
0924 16:12:34.448534 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:34.448539 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:34.448955 (+ 416us) server_negotiation.cc:407] Connection header received
0924 16:12:34.449105 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:34.449109 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:34.449174 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:34.449271 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:34.450660 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:34.451400 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:34.452618 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:34.453026 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:34.453115 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:34.456971 (+ 3856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:34.456976 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:34.456979 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:34.457011 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:34.459506 (+ 2495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:34.460079 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:34.460083 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:34.460085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:34.460208 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:34.460572 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:34.460576 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:34.460578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:34.460937 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:34.461144 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:34.461454 (+ 310us) server_negotiation.cc:299] Negotiation successful
0924 16:12:34.461631 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
W20260924 16:12:34.462510 1317 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:40162
I20260924 16:12:34.464169 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 1382
Sep 24 16:12:34 dist-test-slave-v5j9 krb5kdc[1273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266354, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.29.234.1@KRBTEST.COM
I20260924 16:12:34.480244 1795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:34.465221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37618 (local address 127.29.234.1:38885)
0924 16:12:34.465926 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:34.465933 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:34.465959 (+ 26us) server_negotiation.cc:407] Connection header received
0924 16:12:34.466015 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:34.466022 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:34.466253 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:34.466387 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:34.467520 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:34.468512 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:34.469766 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:34.470320 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:34.470433 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:34.474626 (+ 4193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:34.474637 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:34.474652 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:34.474688 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:34.477635 (+ 2947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:34.478425 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:34.478432 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:34.478434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:34.478551 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:34.478866 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:34.478869 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:34.478871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:34.479213 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:34.479449 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:34.479654 (+ 205us) server_negotiation.cc:299] Negotiation successful
0924 16:12:34.479844 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":389,"threads_started":1}
W20260924 16:12:34.482077 1507 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:37618
W20260924 16:12:34.483192 30632 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 1382: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 16:12:34.483448 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 1382
I20260924 16:12:34.500007 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 1518
Sep 24 16:12:34 dist-test-slave-v5j9 krb5kdc[1273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266354, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.29.234.2@KRBTEST.COM
I20260924 16:12:34.515978 1796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:34.500847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58676 (local address 127.29.234.2:36255)
0924 16:12:34.501432 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:34.501440 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:34.501465 (+ 25us) server_negotiation.cc:407] Connection header received
0924 16:12:34.501527 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:34.501534 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:34.501756 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:34.501890 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:34.503025 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:34.504159 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:34.505512 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:34.506049 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:34.506156 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:34.510356 (+ 4200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:34.510369 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:34.510385 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:34.510415 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:34.513480 (+ 3065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:34.514143 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:34.514150 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:34.514152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:34.514269 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:34.514580 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:34.514583 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:34.514585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:34.514921 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:34.515147 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:34.515379 (+ 232us) server_negotiation.cc:299] Negotiation successful
0924 16:12:34.515580 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":285,"threads_started":1}
W20260924 16:12:34.517493 1644 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:58676
W20260924 16:12:34.518631 30632 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 1518: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 16:12:34.518793 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 1518
I20260924 16:12:34.536722 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 1654
Sep 24 16:12:34 dist-test-slave-v5j9 krb5kdc[1273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266354, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.29.234.3@KRBTEST.COM
I20260924 16:12:34.552475 1797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:34.537710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52616 (local address 127.29.234.3:44799)
0924 16:12:34.538352 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:34.538359 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:34.538386 (+ 27us) server_negotiation.cc:407] Connection header received
0924 16:12:34.538447 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:34.538453 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:34.538665 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:34.538785 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:34.540029 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:34.541064 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:34.542263 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:34.542825 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:34.542927 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:34.547110 (+ 4183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:34.547122 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:34.547140 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:34.547178 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:34.550089 (+ 2911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:34.550692 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:34.550698 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:34.550701 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:34.550813 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:34.551105 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:34.551108 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:34.551110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:34.551432 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:34.551646 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:34.551874 (+ 228us) server_negotiation.cc:299] Negotiation successful
0924 16:12:34.552076 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":333,"threads_started":1}
W20260924 16:12:34.553802 1779 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:52616
W20260924 16:12:34.554924 30632 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 1654: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 16:12:34.555083 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 1654
I20260924 16:12:34.570851 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 1286
I20260924 16:12:34.572096 1349 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:12:34.675215 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 1286
2026-09-24T16:12:34Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3645 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:12:34 dist-test-slave-v5j9 krb5kdc[1801](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:12:34 dist-test-slave-v5j9 krb5kdc[1801](info): set up 2 sockets
Sep 24 16:12:34 dist-test-slave-v5j9 krb5kdc[1801](info): commencing operation
krb5kdc: starting...
W20260924 16:12:35.773007 30632 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Sep 24 16:12:35 dist-test-slave-v5j9 krb5kdc[1801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:12:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:12:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.234.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab.
Entry for principal kudu/127.29.234.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab.
I20260924 16:12:35.970909 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:37931
--webserver_interface=127.29.234.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab
--principal=kudu/127.29.234.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:34601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.234.62:37931
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:12:36.293969 1815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:36.294251 1815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:36.294430 1815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:36.301616 1815 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:12:36.301741 1815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:36.301837 1815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:36.301932 1815 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:12:36.302016 1815 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:12:36.312000 1815 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:34601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.234.62:37931
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab
--principal=kudu/127.29.234.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:37931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.234.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:12:36.313328 1815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:36.314708 1815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:36.326483 1820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:36.326534 1821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:36.326553 1823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:36.327458 1815 server_base.cc:1034] running on GCE node
I20260924 16:12:36.328207 1815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:36.329623 1815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:36.330857 1815 hybrid_clock.cc:648] HybridClock initialized: now 1790266356330820 us; error 57 us; skew 500 ppm
I20260924 16:12:36.331204 1815 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:12:36 dist-test-slave-v5j9 krb5kdc[1801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266356, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:36.344591 1815 init.cc:377] Logged in from keytab as kudu/127.29.234.62@KRBTEST.COM (short username kudu)
I20260924 16:12:36.346382 1815 webserver.cc:460] Webserver started at http://127.29.234.62:36119/ using document root <none> and password file <none>
I20260924 16:12:36.347146 1815 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:36.347293 1815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:36.347635 1815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:36.350744 1815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/minicluster-data/master-0/data/instance:
uuid: "997af1f0b81d40fb84e80f7e93733b19"
format_stamp: "Formatted at 2026-09-24 16:12:36 on dist-test-slave-v5j9"
server_key: "0774b73f3ee55f84d507938e38385035"
server_key_iv: "c56ef3ca252a6e801e6e834f97f07cb3"
server_key_version: "encryptionkey@0"
I20260924 16:12:36.351545 1815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/minicluster-data/master-0/wal/instance:
uuid: "997af1f0b81d40fb84e80f7e93733b19"
format_stamp: "Formatted at 2026-09-24 16:12:36 on dist-test-slave-v5j9"
server_key: "0774b73f3ee55f84d507938e38385035"
server_key_iv: "c56ef3ca252a6e801e6e834f97f07cb3"
server_key_version: "encryptionkey@0"
I20260924 16:12:36.356633 1815 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260924 16:12:36.360052 1830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:36.361474 1815 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:12:36.361673 1815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/minicluster-data/master-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/minicluster-data/master-0/wal
uuid: "997af1f0b81d40fb84e80f7e93733b19"
format_stamp: "Formatted at 2026-09-24 16:12:36 on dist-test-slave-v5j9"
server_key: "0774b73f3ee55f84d507938e38385035"
server_key_iv: "c56ef3ca252a6e801e6e834f97f07cb3"
server_key_version: "encryptionkey@0"
I20260924 16:12:36.361889 1815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:36.401463 1815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:36.405592 1815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:36.405870 1815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:36.424175 1815 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.62:37931
I20260924 16:12:36.424149 1883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.62:37931 every 8 connection(s)
I20260924 16:12:36.425576 1815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
I20260924 16:12:36.429171 1884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:36.430487 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 1815
I20260924 16:12:36.430716 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/minicluster-data/master-0/wal/instance
I20260924 16:12:36.431556 30632 external_mini_cluster.cc:1428] Setting key 2d5e9d1514cf75aeff2db9a412127a1f
I20260924 16:12:36.442747 1884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 997af1f0b81d40fb84e80f7e93733b19: Bootstrap starting.
Sep 24 16:12:36 dist-test-slave-v5j9 krb5kdc[1801](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:12:36.446069 1884 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 997af1f0b81d40fb84e80f7e93733b19: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:36.447244 1884 log.cc:824] T 00000000000000000000000000000000 P 997af1f0b81d40fb84e80f7e93733b19: Log is configured to *not* fsync() on all Append() calls
I20260924 16:12:36.450444 1884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 997af1f0b81d40fb84e80f7e93733b19: No bootstrap required, opened a new log
I20260924 16:12:36.454080 1887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:36.433747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41028 (local address 127.29.234.62:37931)
0924 16:12:36.434539 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:36.434553 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:36.434593 (+ 40us) server_negotiation.cc:407] Connection header received
0924 16:12:36.436170 (+ 1577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:36.436197 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:36.436605 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:36.437000 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:36.438486 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:36.439780 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:36.441467 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:36.442128 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:36.442316 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:36.446652 (+ 4336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:36.446660 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:36.446674 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:36.446711 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:36.449985 (+ 3274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:36.450795 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:36.450802 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:36.450807 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:36.450980 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:36.451393 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:36.451396 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:36.451398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:36.451894 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:36.452115 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:36.452612 (+ 497us) server_negotiation.cc:299] Negotiation successful
0924 16:12:36.453015 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":352,"threads_started":1}
I20260924 16:12:36.458647 1884 raft_consensus.cc:348] T 00000000000000000000000000000000 P 997af1f0b81d40fb84e80f7e93733b19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "997af1f0b81d40fb84e80f7e93733b19" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 37931 } }
I20260924 16:12:36.459060 1884 raft_consensus.cc:374] T 00000000000000000000000000000000 P 997af1f0b81d40fb84e80f7e93733b19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:36.459167 1884 raft_consensus.cc:729] T 00000000000000000000000000000000 P 997af1f0b81d40fb84e80f7e93733b19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 997af1f0b81d40fb84e80f7e93733b19, State: Initialized, Role: FOLLOWER
I20260924 16:12:36.459692 1884 consensus_queue.cc:260] T 00000000000000000000000000000000 P 997af1f0b81d40fb84e80f7e93733b19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "997af1f0b81d40fb84e80f7e93733b19" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 37931 } }
I20260924 16:12:36.459945 1884 raft_consensus.cc:388] T 00000000000000000000000000000000 P 997af1f0b81d40fb84e80f7e93733b19 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:12:36.460085 1884 raft_consensus.cc:482] T 00000000000000000000000000000000 P 997af1f0b81d40fb84e80f7e93733b19 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:12:36.460268 1884 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 997af1f0b81d40fb84e80f7e93733b19 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:36.462461 1884 raft_consensus.cc:504] T 00000000000000000000000000000000 P 997af1f0b81d40fb84e80f7e93733b19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "997af1f0b81d40fb84e80f7e93733b19" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 37931 } }
I20260924 16:12:36.462956 1884 leader_election.cc:302] T 00000000000000000000000000000000 P 997af1f0b81d40fb84e80f7e93733b19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 997af1f0b81d40fb84e80f7e93733b19; no voters:
I20260924 16:12:36.463547 1884 leader_election.cc:288] T 00000000000000000000000000000000 P 997af1f0b81d40fb84e80f7e93733b19 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:12:36.463728 1889 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 997af1f0b81d40fb84e80f7e93733b19 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:12:36.464795 1889 raft_consensus.cc:686] T 00000000000000000000000000000000 P 997af1f0b81d40fb84e80f7e93733b19 [term 1 LEADER]: Becoming Leader. State: Replica: 997af1f0b81d40fb84e80f7e93733b19, State: Running, Role: LEADER
I20260924 16:12:36.465462 1889 consensus_queue.cc:237] T 00000000000000000000000000000000 P 997af1f0b81d40fb84e80f7e93733b19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "997af1f0b81d40fb84e80f7e93733b19" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 37931 } }
I20260924 16:12:36.466363 1884 sys_catalog.cc:564] T 00000000000000000000000000000000 P 997af1f0b81d40fb84e80f7e93733b19 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:12:36.468343 1890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 997af1f0b81d40fb84e80f7e93733b19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "997af1f0b81d40fb84e80f7e93733b19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "997af1f0b81d40fb84e80f7e93733b19" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 37931 } } }
I20260924 16:12:36.468386 1891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 997af1f0b81d40fb84e80f7e93733b19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 997af1f0b81d40fb84e80f7e93733b19. Latest consensus state: current_term: 1 leader_uuid: "997af1f0b81d40fb84e80f7e93733b19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "997af1f0b81d40fb84e80f7e93733b19" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 37931 } } }
I20260924 16:12:36.468714 1890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 997af1f0b81d40fb84e80f7e93733b19 [sys.catalog]: This master's current role is: LEADER
I20260924 16:12:36.468780 1891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 997af1f0b81d40fb84e80f7e93733b19 [sys.catalog]: This master's current role is: LEADER
W20260924 16:12:36.476956 1906 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 997af1f0b81d40fb84e80f7e93733b19: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 16:12:36.477080 1906 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 16:12:36.477214 1895 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:12:36.478624 1895 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:12:36.486119 1895 catalog_manager.cc:1347] Generated new cluster ID: 49f87dd547fb48d1a9985ef3f08423fb
I20260924 16:12:36.486244 1895 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:12:36.505183 1895 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:12:36.506561 1895 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:12:36.513296 1895 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 997af1f0b81d40fb84e80f7e93733b19: Generated new TSK 0
I20260924 16:12:36.514358 1895 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.234.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab.
Entry for principal kudu/127.29.234.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab.
I20260924 16:12:36.573105 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.1:0
--local_ip_for_outbound_sockets=127.29.234.1
--webserver_interface=127.29.234.1
--webserver_port=0
--tserver_master_addrs=127.29.234.62:37931
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab
--principal=kudu/127.29.234.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:34601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:12:36.906491 1911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:36.906764 1911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:36.906925 1911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:36.914311 1911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:36.914439 1911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:36.914620 1911 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.1
I20260924 16:12:36.924602 1911 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:34601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab
--principal=kudu/127.29.234.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.234.1
--webserver_port=0
--tserver_master_addrs=127.29.234.62:37931
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.1
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:12:36.926007 1911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:36.927414 1911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:36.940716 1919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:36.940858 1916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:36.940874 1917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:36.941757 1911 server_base.cc:1034] running on GCE node
I20260924 16:12:36.942396 1911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:36.943405 1911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:36.944628 1911 hybrid_clock.cc:648] HybridClock initialized: now 1790266356944568 us; error 65 us; skew 500 ppm
I20260924 16:12:36.944936 1911 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:12:36 dist-test-slave-v5j9 krb5kdc[1801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266356, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:36.948057 1911 init.cc:377] Logged in from keytab as kudu/127.29.234.1@KRBTEST.COM (short username kudu)
I20260924 16:12:36.949568 1911 webserver.cc:460] Webserver started at http://127.29.234.1:44477/ using document root <none> and password file <none>
I20260924 16:12:36.950232 1911 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:36.950380 1911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:36.950699 1911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:36.953670 1911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/minicluster-data/ts-0/data/instance:
uuid: "580a8eb17f3f41fda30cfc75ffec4d8e"
format_stamp: "Formatted at 2026-09-24 16:12:36 on dist-test-slave-v5j9"
server_key: "e80b27842a2982f0fcbe226f7f9925b9"
server_key_iv: "6b54a15ea74759fe350855de24db74f1"
server_key_version: "encryptionkey@0"
I20260924 16:12:36.954440 1911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/minicluster-data/ts-0/wal/instance:
uuid: "580a8eb17f3f41fda30cfc75ffec4d8e"
format_stamp: "Formatted at 2026-09-24 16:12:36 on dist-test-slave-v5j9"
server_key: "e80b27842a2982f0fcbe226f7f9925b9"
server_key_iv: "6b54a15ea74759fe350855de24db74f1"
server_key_version: "encryptionkey@0"
I20260924 16:12:36.959641 1911 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260924 16:12:36.963034 1926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:36.964368 1911 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 16:12:36.964550 1911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/minicluster-data/ts-0/wal
uuid: "580a8eb17f3f41fda30cfc75ffec4d8e"
format_stamp: "Formatted at 2026-09-24 16:12:36 on dist-test-slave-v5j9"
server_key: "e80b27842a2982f0fcbe226f7f9925b9"
server_key_iv: "6b54a15ea74759fe350855de24db74f1"
server_key_version: "encryptionkey@0"
I20260924 16:12:36.964766 1911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:36.978875 1911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:36.982683 1911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:36.982980 1911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:36.983981 1911 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:12:36.985282 1911 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:12:36.985424 1911 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260924 16:12:36.985574 1911 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:12:36.985649 1911 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:37.019836 1911 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.1:36739
I20260924 16:12:37.019891 2040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.1:36739 every 8 connection(s)
I20260924 16:12:37.021323 1911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
I20260924 16:12:37.024763 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 1911
I20260924 16:12:37.024983 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/minicluster-data/ts-0/wal/instance
I20260924 16:12:37.025905 30632 external_mini_cluster.cc:1428] Setting key c2210dae0003a8dad694084555b30f93
Sep 24 16:12:37 dist-test-slave-v5j9 krb5kdc[1801](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266356, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.1@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:12:37.045387 2043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:37.024152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.1:42887 (local address 127.29.234.62:37931)
0924 16:12:37.024671 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:37.024677 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:37.026602 (+ 1925us) server_negotiation.cc:407] Connection header received
0924 16:12:37.028408 (+ 1806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:37.028413 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:37.028480 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:37.028566 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:37.030788 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:37.031481 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:37.034004 (+ 2523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:37.034638 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:37.034775 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:37.038293 (+ 3518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:37.038302 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:37.038310 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:37.038352 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:37.041969 (+ 3617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:37.042724 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:37.042733 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:37.042740 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:37.042917 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:37.043384 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:37.043391 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:37.043396 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:37.043794 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:37.043976 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:37.044658 (+ 682us) server_negotiation.cc:299] Negotiation successful
0924 16:12:37.044926 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":295,"threads_started":1}
I20260924 16:12:37.046171 2044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:37.024865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:37931 (local address 127.29.234.1:42887)
0924 16:12:37.025860 (+ 995us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:37.025912 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:37.028105 (+ 2193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:37.028865 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:37.028879 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:37.029727 (+ 848us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:37.030604 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:37.030631 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:37.031643 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:37.031652 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:37.033867 (+ 2215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:37.033886 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:37.034851 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:37.034855 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:37.035048 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:37.035750 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:37.035780 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:37.038055 (+ 2275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:37.042124 (+ 4069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:37.042132 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:37.042146 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:37.042578 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:37.043041 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:37.043044 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:37.043047 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:37.043255 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:37.043924 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:37.043930 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:37.044269 (+ 339us) client_negotiation.cc:769] Sending connection context
0924 16:12:37.044494 (+ 225us) client_negotiation.cc:240] Negotiation successful
0924 16:12:37.044911 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":393,"threads_started":1}
WARNING: no policy specified for kudu/127.29.234.2@KRBTEST.COM; defaulting to no policy
I20260924 16:12:37.047923 2041 heartbeater.cc:347] Connected to a master server at 127.29.234.62:37931
I20260924 16:12:37.048265 2041 heartbeater.cc:464] Registering TS with master...
I20260924 16:12:37.049062 2041 heartbeater.cc:511] Master 127.29.234.62:37931 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.2@KRBTEST.COM" created.
I20260924 16:12:37.051476 1847 ts_manager.cc:194] Registered new tserver with Master: 580a8eb17f3f41fda30cfc75ffec4d8e (127.29.234.1:36739)
I20260924 16:12:37.052992 1847 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.234.1@KRBTEST.COM'} at 127.29.234.1:42887
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab.
Entry for principal kudu/127.29.234.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab.
I20260924 16:12:37.074388 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.2:0
--local_ip_for_outbound_sockets=127.29.234.2
--webserver_interface=127.29.234.2
--webserver_port=0
--tserver_master_addrs=127.29.234.62:37931
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab
--principal=kudu/127.29.234.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:34601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:12:37.397459 2047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:37.397753 2047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:37.397917 2047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:37.404933 2047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:37.405076 2047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:37.405261 2047 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.2
I20260924 16:12:37.414866 2047 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:34601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab
--principal=kudu/127.29.234.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.234.2
--webserver_port=0
--tserver_master_addrs=127.29.234.62:37931
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.2
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:12:37.416112 2047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:37.417459 2047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:37.432044 2052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:37.432119 2053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:37.432364 2055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:37.432888 2047 server_base.cc:1034] running on GCE node
I20260924 16:12:37.433559 2047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:37.434590 2047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:37.435799 2047 hybrid_clock.cc:648] HybridClock initialized: now 1790266357435742 us; error 54 us; skew 500 ppm
I20260924 16:12:37.436108 2047 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:12:37 dist-test-slave-v5j9 krb5kdc[1801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266357, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:37.439632 2047 init.cc:377] Logged in from keytab as kudu/127.29.234.2@KRBTEST.COM (short username kudu)
I20260924 16:12:37.441056 2047 webserver.cc:460] Webserver started at http://127.29.234.2:33219/ using document root <none> and password file <none>
I20260924 16:12:37.441774 2047 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:37.441915 2047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:37.442214 2047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:37.444960 2047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/minicluster-data/ts-1/data/instance:
uuid: "7a1e3ed454b94442af1c386989c83f38"
format_stamp: "Formatted at 2026-09-24 16:12:37 on dist-test-slave-v5j9"
server_key: "efb2e52aaecc31e8f6e65ed972b32ed7"
server_key_iv: "6c3d833437f8602068a852b7c842a007"
server_key_version: "encryptionkey@0"
I20260924 16:12:37.445746 2047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/minicluster-data/ts-1/wal/instance:
uuid: "7a1e3ed454b94442af1c386989c83f38"
format_stamp: "Formatted at 2026-09-24 16:12:37 on dist-test-slave-v5j9"
server_key: "efb2e52aaecc31e8f6e65ed972b32ed7"
server_key_iv: "6c3d833437f8602068a852b7c842a007"
server_key_version: "encryptionkey@0"
I20260924 16:12:37.450732 2047 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 16:12:37.454056 2062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:37.455250 2047 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:12:37.455417 2047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/minicluster-data/ts-1/wal
uuid: "7a1e3ed454b94442af1c386989c83f38"
format_stamp: "Formatted at 2026-09-24 16:12:37 on dist-test-slave-v5j9"
server_key: "efb2e52aaecc31e8f6e65ed972b32ed7"
server_key_iv: "6c3d833437f8602068a852b7c842a007"
server_key_version: "encryptionkey@0"
I20260924 16:12:37.455612 2047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:37.483265 2047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:37.487684 2047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:37.487988 2047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:37.488925 2047 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:12:37.490340 2047 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:12:37.490443 2047 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:37.490587 2047 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:12:37.490669 2047 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:37.523284 2047 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.2:43243
I20260924 16:12:37.523335 2177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.2:43243 every 8 connection(s)
I20260924 16:12:37.524619 2047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
I20260924 16:12:37.532819 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 2047
I20260924 16:12:37.533008 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/minicluster-data/ts-1/wal/instance
I20260924 16:12:37.533583 30632 external_mini_cluster.cc:1428] Setting key c598cf0084e61bc2dccc74f3589904fd
Sep 24 16:12:37 dist-test-slave-v5j9 krb5kdc[1801](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266357, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.2@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:12:37.545748 2043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:37.527380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.2:53519 (local address 127.29.234.62:37931)
0924 16:12:37.527555 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:37.527559 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:37.528662 (+ 1103us) server_negotiation.cc:407] Connection header received
0924 16:12:37.530566 (+ 1904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:37.530573 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:37.530656 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:37.530766 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:37.532166 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:37.533185 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:37.535023 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:37.535619 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:37.535761 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:37.539745 (+ 3984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:37.539750 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:37.539754 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:37.539784 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:37.542826 (+ 3042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:37.543545 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:37.543550 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:37.543552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:37.543666 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:37.544086 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:37.544091 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:37.544096 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:37.544379 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:37.544528 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:37.545170 (+ 642us) server_negotiation.cc:299] Negotiation successful
0924 16:12:37.545423 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 16:12:37.546737 2180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:37.527853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:37931 (local address 127.29.234.2:53519)
0924 16:12:37.528511 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:37.528553 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:37.530302 (+ 1749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:37.531000 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:37.531007 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:37.531489 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:37.532009 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:37.532024 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:37.533382 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:37.533389 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:37.534872 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:37.534887 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:37.535861 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:37.535869 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:37.536165 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:37.536875 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:37.536909 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:37.539517 (+ 2608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:37.542972 (+ 3455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:37.542981 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:37.542996 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:37.543412 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:37.543774 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:37.543779 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:37.543783 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:37.543969 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:37.544501 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:37.544509 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:37.544822 (+ 313us) client_negotiation.cc:769] Sending connection context
0924 16:12:37.545071 (+ 249us) client_negotiation.cc:240] Negotiation successful
0924 16:12:37.545601 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":258,"threads_started":1}
I20260924 16:12:37.548164 2178 heartbeater.cc:347] Connected to a master server at 127.29.234.62:37931
I20260924 16:12:37.548451 2178 heartbeater.cc:464] Registering TS with master...
I20260924 16:12:37.549149 2178 heartbeater.cc:511] Master 127.29.234.62:37931 requested a full tablet report, sending...
I20260924 16:12:37.550722 1847 ts_manager.cc:194] Registered new tserver with Master: 7a1e3ed454b94442af1c386989c83f38 (127.29.234.2:43243)
I20260924 16:12:37.551947 1847 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.234.2@KRBTEST.COM'} at 127.29.234.2:53519
WARNING: no policy specified for kudu/127.29.234.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab.
Entry for principal kudu/127.29.234.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab.
I20260924 16:12:37.577942 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.3:0
--local_ip_for_outbound_sockets=127.29.234.3
--webserver_interface=127.29.234.3
--webserver_port=0
--tserver_master_addrs=127.29.234.62:37931
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab
--principal=kudu/127.29.234.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:34601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:12:37.900043 2183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:37.900353 2183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:37.900512 2183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:37.907503 2183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:37.907634 2183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:37.907799 2183 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.3
I20260924 16:12:37.917714 2183 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:34601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab
--principal=kudu/127.29.234.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.234.3
--webserver_port=0
--tserver_master_addrs=127.29.234.62:37931
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.3
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:12:37.919008 2183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:37.920296 2183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:37.933018 2188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:37.933061 2189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:37.933110 2191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:37.933795 2183 server_base.cc:1034] running on GCE node
I20260924 16:12:37.934404 2183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:37.935282 2183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:37.936527 2183 hybrid_clock.cc:648] HybridClock initialized: now 1790266357936474 us; error 55 us; skew 500 ppm
I20260924 16:12:37.936837 2183 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:12:37 dist-test-slave-v5j9 krb5kdc[1801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266357, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:37.939970 2183 init.cc:377] Logged in from keytab as kudu/127.29.234.3@KRBTEST.COM (short username kudu)
I20260924 16:12:37.941411 2183 webserver.cc:460] Webserver started at http://127.29.234.3:42307/ using document root <none> and password file <none>
I20260924 16:12:37.942094 2183 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:37.942221 2183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:37.942539 2183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:37.945453 2183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/minicluster-data/ts-2/data/instance:
uuid: "18add7fb5e3443fa93ee54d7464f3b78"
format_stamp: "Formatted at 2026-09-24 16:12:37 on dist-test-slave-v5j9"
server_key: "e732e1417ca7818667971874a5434b88"
server_key_iv: "5f9621fd1f7b844c6284eb7649c66240"
server_key_version: "encryptionkey@0"
I20260924 16:12:37.946231 2183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/minicluster-data/ts-2/wal/instance:
uuid: "18add7fb5e3443fa93ee54d7464f3b78"
format_stamp: "Formatted at 2026-09-24 16:12:37 on dist-test-slave-v5j9"
server_key: "e732e1417ca7818667971874a5434b88"
server_key_iv: "5f9621fd1f7b844c6284eb7649c66240"
server_key_version: "encryptionkey@0"
I20260924 16:12:37.951399 2183 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260924 16:12:37.954835 2198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:37.956158 2183 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260924 16:12:37.956351 2183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/minicluster-data/ts-2/wal
uuid: "18add7fb5e3443fa93ee54d7464f3b78"
format_stamp: "Formatted at 2026-09-24 16:12:37 on dist-test-slave-v5j9"
server_key: "e732e1417ca7818667971874a5434b88"
server_key_iv: "5f9621fd1f7b844c6284eb7649c66240"
server_key_version: "encryptionkey@0"
I20260924 16:12:37.956544 2183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:37.972371 2183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:37.976675 2183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:37.976974 2183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:37.977948 2183 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:12:37.979354 2183 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:12:37.979454 2183 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:37.979584 2183 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:12:37.979647 2183 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:38.010123 2183 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.3:42361
I20260924 16:12:38.010161 2313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.3:42361 every 8 connection(s)
I20260924 16:12:38.011477 2183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
I20260924 16:12:38.016324 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 2183
I20260924 16:12:38.016515 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790266323169505-30632-0/minicluster-data/ts-2/wal/instance
I20260924 16:12:38.016963 30632 external_mini_cluster.cc:1428] Setting key cd18cb6b568dabac4dbd325e8f6961a2
Sep 24 16:12:38 dist-test-slave-v5j9 krb5kdc[1801](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266357, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.3@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:12:38.032843 2043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:38.014092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.3:60943 (local address 127.29.234.62:37931)
0924 16:12:38.014307 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:38.014313 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:38.015594 (+ 1281us) server_negotiation.cc:407] Connection header received
0924 16:12:38.017744 (+ 2150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:38.017751 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:38.017845 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:38.017968 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:38.020146 (+ 2178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:38.020869 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:38.023075 (+ 2206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:38.023549 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:38.023639 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:38.027280 (+ 3641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:38.027285 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:38.027288 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:38.027318 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:38.029872 (+ 2554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:38.030551 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:38.030561 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:38.030564 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:38.030707 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:38.031134 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:38.031144 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:38.031146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:38.031413 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:38.031583 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:38.032277 (+ 694us) server_negotiation.cc:299] Negotiation successful
0924 16:12:38.032490 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 16:12:38.033859 2316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:38.014691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:37931 (local address 127.29.234.3:60943)
0924 16:12:38.015433 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:38.015477 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:38.017311 (+ 1834us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:38.018442 (+ 1131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:38.018451 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:38.019182 (+ 731us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:38.019967 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:38.019993 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:38.021018 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:38.021024 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:38.022945 (+ 1921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:38.022962 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:38.023770 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:38.023777 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:38.024076 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:38.024761 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:38.024790 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:38.027069 (+ 2279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:38.029986 (+ 2917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:38.029992 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:38.030006 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:38.030416 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:38.030819 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:38.030825 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:38.030829 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:38.031014 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:38.031552 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:38.031560 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:38.031907 (+ 347us) client_negotiation.cc:769] Sending connection context
0924 16:12:38.032162 (+ 255us) client_negotiation.cc:240] Negotiation successful
0924 16:12:38.032657 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":279,"threads_started":1}
I20260924 16:12:38.035243 2314 heartbeater.cc:347] Connected to a master server at 127.29.234.62:37931
I20260924 16:12:38.035552 2314 heartbeater.cc:464] Registering TS with master...
I20260924 16:12:38.036258 2314 heartbeater.cc:511] Master 127.29.234.62:37931 requested a full tablet report, sending...
I20260924 16:12:38.037707 1847 ts_manager.cc:194] Registered new tserver with Master: 18add7fb5e3443fa93ee54d7464f3b78 (127.29.234.3:42361)
I20260924 16:12:38.038535 1847 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.234.3@KRBTEST.COM'} at 127.29.234.3:60943
I20260924 16:12:38.044138 30632 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:12:38.056130 2041 heartbeater.cc:503] Master 127.29.234.62:37931 was elected leader, sending a full tablet report...
Sep 24 16:12:38 dist-test-slave-v5j9 krb5kdc[1801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 16:12:38 dist-test-slave-v5j9 krb5kdc[1801](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.234.1@KRBTEST.COM
I20260924 16:12:38.092600 2321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:38.078012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44616 (local address 127.29.234.1:36739)
0924 16:12:38.078615 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:38.078623 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:38.078653 (+ 30us) server_negotiation.cc:407] Connection header received
0924 16:12:38.078716 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:38.078723 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:38.078942 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:38.079074 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:38.080400 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:38.081490 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:38.082621 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:38.083210 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:38.083304 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:38.086986 (+ 3682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:38.086996 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:38.087018 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:38.087053 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:38.090040 (+ 2987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:38.090668 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:38.090677 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:38.090679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:38.090811 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:38.091127 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:38.091130 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:38.091132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:38.091473 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:38.091721 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:38.091958 (+ 237us) server_negotiation.cc:299] Negotiation successful
0924 16:12:38.092138 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":315,"threads_started":1}
Sep 24 16:12:38 dist-test-slave-v5j9 krb5kdc[1801](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:12:38.109396 2043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:38.096870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41044 (local address 127.29.234.62:37931)
0924 16:12:38.097087 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:38.097092 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:38.097154 (+ 62us) server_negotiation.cc:407] Connection header received
0924 16:12:38.097288 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:38.097292 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:38.097404 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:38.097484 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:38.098680 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:38.099420 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:38.100560 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:38.100968 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:38.101058 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:38.104679 (+ 3621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:38.104684 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:38.104687 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:38.104718 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:38.107139 (+ 2421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:38.107682 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:38.107686 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:38.107689 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:38.107829 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:38.108123 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:38.108126 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:38.108128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:38.108433 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:38.108610 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:38.108850 (+ 240us) server_negotiation.cc:299] Negotiation successful
0924 16:12:38.109008 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260924 16:12:38.110015 1847 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41044
I20260924 16:12:38.111027 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 1911
I20260924 16:12:38.123387 2321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:38.111850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44624 (local address 127.29.234.1:36739)
0924 16:12:38.112071 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:38.112078 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:38.112306 (+ 228us) server_negotiation.cc:407] Connection header received
0924 16:12:38.112490 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:38.112501 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:38.112563 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:38.112666 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:38.113920 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:38.114599 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:38.115887 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:38.116300 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:38.116430 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:38.118546 (+ 2116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:38.118551 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:38.118554 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:38.118583 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:38.120884 (+ 2301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:38.121565 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:38.121571 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:38.121575 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:38.121699 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:38.122077 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:38.122082 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:38.122085 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:38.122322 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:38.122565 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:38.122868 (+ 303us) server_negotiation.cc:299] Negotiation successful
0924 16:12:38.123074 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 16:12:38.123963 2036 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:12:38.205691 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 1911
I20260924 16:12:38.222769 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 2047
Sep 24 16:12:38 dist-test-slave-v5j9 krb5kdc[1801](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.234.2@KRBTEST.COM
I20260924 16:12:38.238538 2326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:38.223747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45912 (local address 127.29.234.2:43243)
0924 16:12:38.224264 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:38.224271 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:38.224297 (+ 26us) server_negotiation.cc:407] Connection header received
0924 16:12:38.224359 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:38.224366 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:38.224589 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:38.224779 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:38.226068 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:38.227075 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:38.228392 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:38.228942 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:38.229039 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:38.233205 (+ 4166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:38.233216 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:38.233231 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:38.233263 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:38.235996 (+ 2733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:38.236638 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:38.236645 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:38.236647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:38.236757 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:38.237077 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:38.237080 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:38.237081 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:38.237506 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:38.237739 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:38.238044 (+ 305us) server_negotiation.cc:299] Negotiation successful
0924 16:12:38.238237 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":249,"threads_started":1}
I20260924 16:12:38.239786 2173 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:12:38.323670 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 2047
I20260924 16:12:38.340564 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 2183
Sep 24 16:12:38 dist-test-slave-v5j9 krb5kdc[1801](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.234.3@KRBTEST.COM
I20260924 16:12:38.356386 2328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:38.341570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57918 (local address 127.29.234.3:42361)
0924 16:12:38.342181 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:38.342188 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:38.342218 (+ 30us) server_negotiation.cc:407] Connection header received
0924 16:12:38.342277 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:38.342283 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:38.342513 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:38.342639 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:38.343798 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:38.344803 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:38.346135 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:38.346691 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:38.346788 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:38.350974 (+ 4186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:38.350986 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:38.351001 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:38.351031 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:38.353987 (+ 2956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:38.354628 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:38.354636 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:38.354638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:38.354749 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:38.355041 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:38.355044 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:38.355046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:38.355377 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:38.355606 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:38.355867 (+ 261us) server_negotiation.cc:299] Negotiation successful
0924 16:12:38.356065 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":312,"threads_started":1}
I20260924 16:12:38.357879 2309 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:12:38.442509 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 2183
I20260924 16:12:38.458619 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 1815
I20260924 16:12:38.459861 1879 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:12:38.564585 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 1815
2026-09-24T16:12:38Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3889 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:12:38 dist-test-slave-v5j9 krb5kdc[2333](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:12:38 dist-test-slave-v5j9 krb5kdc[2333](info): set up 2 sockets
Sep 24 16:12:38 dist-test-slave-v5j9 krb5kdc[2333](info): commencing operation
krb5kdc: starting...
W20260924 16:12:39.661058 30632 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Sep 24 16:12:39 dist-test-slave-v5j9 krb5kdc[2333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:12:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:12:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.234.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab.
Entry for principal kudu/127.29.234.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab.
I20260924 16:12:39.860728 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:34777
--webserver_interface=127.29.234.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab
--principal=kudu/127.29.234.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:45593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.234.62:34777
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:12:40.182830 2346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:40.183174 2346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:40.183343 2346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:40.190441 2346 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:12:40.190596 2346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:40.190696 2346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:40.190795 2346 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:12:40.190878 2346 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:12:40.200589 2346 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:45593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.234.62:34777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab
--principal=kudu/127.29.234.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:34777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.234.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:12:40.201962 2346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:40.203321 2346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:40.215237 2354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:40.215391 2352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:40.215265 2351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:40.215951 2346 server_base.cc:1034] running on GCE node
I20260924 16:12:40.216876 2346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:40.218286 2346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:40.219560 2346 hybrid_clock.cc:648] HybridClock initialized: now 1790266360219504 us; error 57 us; skew 500 ppm
I20260924 16:12:40.219905 2346 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:12:40 dist-test-slave-v5j9 krb5kdc[2333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266360, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:40.230981 2346 init.cc:377] Logged in from keytab as kudu/127.29.234.62@KRBTEST.COM (short username kudu)
I20260924 16:12:40.232161 2346 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:40.232259 2346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:40.232578 2346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:40.235638 2346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/minicluster-data/master-0/data/instance:
uuid: "66c70841ea7d4421b09018f49633fb95"
format_stamp: "Formatted at 2026-09-24 16:12:40 on dist-test-slave-v5j9"
server_key: "f0f5d390834cf11dcd6bd641bc52b81a"
server_key_iv: "32604663b09a7d3f2d9cb9e6b77d3a49"
server_key_version: "encryptionkey@0"
I20260924 16:12:40.236382 2346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/minicluster-data/master-0/wal/instance:
uuid: "66c70841ea7d4421b09018f49633fb95"
format_stamp: "Formatted at 2026-09-24 16:12:40 on dist-test-slave-v5j9"
server_key: "f0f5d390834cf11dcd6bd641bc52b81a"
server_key_iv: "32604663b09a7d3f2d9cb9e6b77d3a49"
server_key_version: "encryptionkey@0"
I20260924 16:12:40.241577 2346 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260924 16:12:40.244874 2360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:40.246253 2346 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:12:40.246387 2346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/minicluster-data/master-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/minicluster-data/master-0/wal
uuid: "66c70841ea7d4421b09018f49633fb95"
format_stamp: "Formatted at 2026-09-24 16:12:40 on dist-test-slave-v5j9"
server_key: "f0f5d390834cf11dcd6bd641bc52b81a"
server_key_iv: "32604663b09a7d3f2d9cb9e6b77d3a49"
server_key_version: "encryptionkey@0"
I20260924 16:12:40.246564 2346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:40.263546 2346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:40.269309 2346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:40.269631 2346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:40.288496 2346 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.62:34777
I20260924 16:12:40.288509 2412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.62:34777 every 8 connection(s)
I20260924 16:12:40.289570 2346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
I20260924 16:12:40.293273 2413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:40.299944 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 2346
I20260924 16:12:40.300237 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/minicluster-data/master-0/wal/instance
I20260924 16:12:40.300886 30632 external_mini_cluster.cc:1428] Setting key dadff9baa966db37e741fc6b96789230
I20260924 16:12:40.304076 2413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66c70841ea7d4421b09018f49633fb95: Bootstrap starting.
I20260924 16:12:40.307682 2413 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66c70841ea7d4421b09018f49633fb95: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:40.308898 2413 log.cc:824] T 00000000000000000000000000000000 P 66c70841ea7d4421b09018f49633fb95: Log is configured to *not* fsync() on all Append() calls
I20260924 16:12:40.312162 2413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66c70841ea7d4421b09018f49633fb95: No bootstrap required, opened a new log
Sep 24 16:12:40 dist-test-slave-v5j9 krb5kdc[2333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:12:40.321460 2413 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66c70841ea7d4421b09018f49633fb95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66c70841ea7d4421b09018f49633fb95" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 34777 } }
I20260924 16:12:40.321902 2413 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66c70841ea7d4421b09018f49633fb95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:40.322053 2413 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66c70841ea7d4421b09018f49633fb95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66c70841ea7d4421b09018f49633fb95, State: Initialized, Role: FOLLOWER
I20260924 16:12:40.322865 2413 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66c70841ea7d4421b09018f49633fb95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66c70841ea7d4421b09018f49633fb95" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 34777 } }
I20260924 16:12:40.323163 2413 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66c70841ea7d4421b09018f49633fb95 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:12:40.323331 2413 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66c70841ea7d4421b09018f49633fb95 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:12:40.323521 2413 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66c70841ea7d4421b09018f49633fb95 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:40.326702 2413 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66c70841ea7d4421b09018f49633fb95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66c70841ea7d4421b09018f49633fb95" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 34777 } }
I20260924 16:12:40.327323 2413 leader_election.cc:302] T 00000000000000000000000000000000 P 66c70841ea7d4421b09018f49633fb95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66c70841ea7d4421b09018f49633fb95; no voters:
I20260924 16:12:40.328051 2413 leader_election.cc:288] T 00000000000000000000000000000000 P 66c70841ea7d4421b09018f49633fb95 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:12:40.328197 2418 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66c70841ea7d4421b09018f49633fb95 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:12:40.329891 2416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:40.303125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51006 (local address 127.29.234.62:34777)
0924 16:12:40.303927 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:40.303940 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:40.303983 (+ 43us) server_negotiation.cc:407] Connection header received
0924 16:12:40.305790 (+ 1807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:40.305812 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:40.306210 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:40.306518 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:40.308449 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:40.310303 (+ 1854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:40.312093 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:40.312853 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:40.312994 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:40.318660 (+ 5666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:40.318669 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:40.318690 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:40.318730 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:40.324202 (+ 5472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:40.324943 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:40.324952 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:40.324962 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:40.325183 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:40.325626 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:40.325633 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:40.325640 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:40.326403 (+ 763us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:40.326741 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:40.327283 (+ 542us) server_negotiation.cc:299] Negotiation successful
0924 16:12:40.327831 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":370,"threads_started":1}
I20260924 16:12:40.330430 2413 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66c70841ea7d4421b09018f49633fb95 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:12:40.332603 2418 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66c70841ea7d4421b09018f49633fb95 [term 1 LEADER]: Becoming Leader. State: Replica: 66c70841ea7d4421b09018f49633fb95, State: Running, Role: LEADER
I20260924 16:12:40.333379 2418 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66c70841ea7d4421b09018f49633fb95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66c70841ea7d4421b09018f49633fb95" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 34777 } }
I20260924 16:12:40.336968 2419 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66c70841ea7d4421b09018f49633fb95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66c70841ea7d4421b09018f49633fb95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66c70841ea7d4421b09018f49633fb95" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 34777 } } }
I20260924 16:12:40.337417 2419 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66c70841ea7d4421b09018f49633fb95 [sys.catalog]: This master's current role is: LEADER
I20260924 16:12:40.339115 2420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66c70841ea7d4421b09018f49633fb95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66c70841ea7d4421b09018f49633fb95. Latest consensus state: current_term: 1 leader_uuid: "66c70841ea7d4421b09018f49633fb95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66c70841ea7d4421b09018f49633fb95" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 34777 } } }
I20260924 16:12:40.339344 2420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66c70841ea7d4421b09018f49633fb95 [sys.catalog]: This master's current role is: LEADER
W20260924 16:12:40.342819 2432 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 66c70841ea7d4421b09018f49633fb95: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 16:12:40.342981 2432 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 16:12:40.343145 2433 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:12:40.344660 2433 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:12:40.353295 2433 catalog_manager.cc:1347] Generated new cluster ID: f40cb0c2d4dd43bc8832965c7ef6a8fb
I20260924 16:12:40.353475 2433 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:12:40.372967 2433 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:12:40.375100 2433 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:12:40.388284 2433 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 66c70841ea7d4421b09018f49633fb95: Generated new TSK 0
I20260924 16:12:40.389725 2433 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.234.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab.
Entry for principal kudu/127.29.234.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab.
I20260924 16:12:40.453008 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.1:0
--local_ip_for_outbound_sockets=127.29.234.1
--webserver_interface=127.29.234.1
--webserver_port=0
--tserver_master_addrs=127.29.234.62:34777
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab
--principal=kudu/127.29.234.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:45593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:12:40.783732 2440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:40.784055 2440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:40.784228 2440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:40.791465 2440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:40.791632 2440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:40.791822 2440 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.1
I20260924 16:12:40.801786 2440 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:45593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab
--principal=kudu/127.29.234.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.234.1
--webserver_port=0
--tserver_master_addrs=127.29.234.62:34777
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.1
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:12:40.803195 2440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:40.804569 2440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:40.818120 2448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:40.818156 2445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:40.818569 2446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:40.818944 2440 server_base.cc:1034] running on GCE node
I20260924 16:12:40.819577 2440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:40.820516 2440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:40.821794 2440 hybrid_clock.cc:648] HybridClock initialized: now 1790266360821744 us; error 47 us; skew 500 ppm
I20260924 16:12:40.822134 2440 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:12:40 dist-test-slave-v5j9 krb5kdc[2333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266360, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:40.826267 2440 init.cc:377] Logged in from keytab as kudu/127.29.234.1@KRBTEST.COM (short username kudu)
I20260924 16:12:40.827438 2440 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:40.827571 2440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:40.827895 2440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:40.831101 2440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/minicluster-data/ts-0/data/instance:
uuid: "b77397a5eec341d2b526a5c6df0b520f"
format_stamp: "Formatted at 2026-09-24 16:12:40 on dist-test-slave-v5j9"
server_key: "57fc6528be63f720c5ef36ae2a6f9a3f"
server_key_iv: "2b166d717117c8fb3fc564d931f71ab3"
server_key_version: "encryptionkey@0"
I20260924 16:12:40.832070 2440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/minicluster-data/ts-0/wal/instance:
uuid: "b77397a5eec341d2b526a5c6df0b520f"
format_stamp: "Formatted at 2026-09-24 16:12:40 on dist-test-slave-v5j9"
server_key: "57fc6528be63f720c5ef36ae2a6f9a3f"
server_key_iv: "2b166d717117c8fb3fc564d931f71ab3"
server_key_version: "encryptionkey@0"
I20260924 16:12:40.837473 2440 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 16:12:40.841070 2454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:40.842381 2440 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260924 16:12:40.842562 2440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/minicluster-data/ts-0/wal
uuid: "b77397a5eec341d2b526a5c6df0b520f"
format_stamp: "Formatted at 2026-09-24 16:12:40 on dist-test-slave-v5j9"
server_key: "57fc6528be63f720c5ef36ae2a6f9a3f"
server_key_iv: "2b166d717117c8fb3fc564d931f71ab3"
server_key_version: "encryptionkey@0"
I20260924 16:12:40.842775 2440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:40.867278 2440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:40.874187 2440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:40.874539 2440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:40.875545 2440 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:12:40.876943 2440 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:12:40.877065 2440 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:40.877195 2440 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:12:40.877282 2440 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:40.910986 2440 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.1:37255
I20260924 16:12:40.911070 2567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.1:37255 every 8 connection(s)
I20260924 16:12:40.911886 2440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
I20260924 16:12:40.914139 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 2440
I20260924 16:12:40.914386 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/minicluster-data/ts-0/wal/instance
I20260924 16:12:40.915215 30632 external_mini_cluster.cc:1428] Setting key 7dd64f029449dd0aefc51c840045b015
Sep 24 16:12:40 dist-test-slave-v5j9 krb5kdc[2333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266360, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.1@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.234.2@KRBTEST.COM; defaulting to no policy
I20260924 16:12:40.936439 2570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:40.914651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.1:34659 (local address 127.29.234.62:34777)
0924 16:12:40.915118 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:40.915122 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:40.916444 (+ 1322us) server_negotiation.cc:407] Connection header received
0924 16:12:40.918584 (+ 2140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:40.918590 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:40.918680 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:40.918793 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:40.920311 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:40.921130 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:40.923335 (+ 2205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:40.923722 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:40.923819 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:40.928619 (+ 4800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:40.928624 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:40.928627 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:40.928659 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:40.933139 (+ 4480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:40.934044 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:40.934050 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:40.934054 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:40.934217 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:40.934701 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:40.934706 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:40.934710 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:40.934978 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:40.935129 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:40.935831 (+ 702us) server_negotiation.cc:299] Negotiation successful
0924 16:12:40.936070 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":254,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.2@KRBTEST.COM" created.
I20260924 16:12:40.937628 2571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:40.915374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:34777 (local address 127.29.234.1:34659)
0924 16:12:40.916258 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:40.916300 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:40.918311 (+ 2011us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:40.919034 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:40.919043 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:40.919556 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:40.920129 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:40.920146 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:40.921296 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:40.921300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:40.923191 (+ 1891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:40.923206 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:40.923940 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:40.923950 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:40.924274 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:40.925762 (+ 1488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:40.925797 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:40.928401 (+ 2604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:40.933281 (+ 4880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:40.933289 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:40.933306 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:40.933920 (+ 614us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:40.934322 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:40.934327 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:40.934331 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:40.934576 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:40.935106 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:40.935115 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:40.935460 (+ 345us) client_negotiation.cc:769] Sending connection context
0924 16:12:40.935733 (+ 273us) client_negotiation.cc:240] Negotiation successful
0924 16:12:40.936251 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":359,"threads_started":1}
I20260924 16:12:40.939268 2568 heartbeater.cc:347] Connected to a master server at 127.29.234.62:34777
I20260924 16:12:40.939575 2568 heartbeater.cc:464] Registering TS with master...
I20260924 16:12:40.940227 2568 heartbeater.cc:511] Master 127.29.234.62:34777 requested a full tablet report, sending...
I20260924 16:12:40.942519 2377 ts_manager.cc:194] Registered new tserver with Master: b77397a5eec341d2b526a5c6df0b520f (127.29.234.1:37255)
I20260924 16:12:40.944156 2377 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.234.1@KRBTEST.COM'} at 127.29.234.1:34659
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab.
Entry for principal kudu/127.29.234.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab.
I20260924 16:12:40.963864 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.2:0
--local_ip_for_outbound_sockets=127.29.234.2
--webserver_interface=127.29.234.2
--webserver_port=0
--tserver_master_addrs=127.29.234.62:34777
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab
--principal=kudu/127.29.234.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:45593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:12:41.288501 2574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:41.288776 2574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:41.288936 2574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:41.296154 2574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:41.296264 2574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:41.296446 2574 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.2
I20260924 16:12:41.306063 2574 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:45593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab
--principal=kudu/127.29.234.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.234.2
--webserver_port=0
--tserver_master_addrs=127.29.234.62:34777
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.2
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:12:41.307292 2574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:41.308614 2574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:41.321848 2580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:41.321862 2579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:41.322093 2582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:41.322594 2574 server_base.cc:1034] running on GCE node
I20260924 16:12:41.323374 2574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:41.324337 2574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:41.325560 2574 hybrid_clock.cc:648] HybridClock initialized: now 1790266361325510 us; error 43 us; skew 500 ppm
I20260924 16:12:41.325889 2574 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:12:41 dist-test-slave-v5j9 krb5kdc[2333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266361, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:41.329839 2574 init.cc:377] Logged in from keytab as kudu/127.29.234.2@KRBTEST.COM (short username kudu)
I20260924 16:12:41.330986 2574 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:41.331070 2574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:41.331382 2574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:41.334605 2574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/minicluster-data/ts-1/data/instance:
uuid: "e3e1eeaa41064392900d85f3cde27aba"
format_stamp: "Formatted at 2026-09-24 16:12:41 on dist-test-slave-v5j9"
server_key: "267c985a25abeeb624ceded28d28fa5f"
server_key_iv: "812f823361d0db027473a62153cef637"
server_key_version: "encryptionkey@0"
I20260924 16:12:41.335372 2574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/minicluster-data/ts-1/wal/instance:
uuid: "e3e1eeaa41064392900d85f3cde27aba"
format_stamp: "Formatted at 2026-09-24 16:12:41 on dist-test-slave-v5j9"
server_key: "267c985a25abeeb624ceded28d28fa5f"
server_key_iv: "812f823361d0db027473a62153cef637"
server_key_version: "encryptionkey@0"
I20260924 16:12:41.340453 2574 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 16:12:41.343990 2590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:41.345479 2574 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:12:41.345669 2574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/minicluster-data/ts-1/wal
uuid: "e3e1eeaa41064392900d85f3cde27aba"
format_stamp: "Formatted at 2026-09-24 16:12:41 on dist-test-slave-v5j9"
server_key: "267c985a25abeeb624ceded28d28fa5f"
server_key_iv: "812f823361d0db027473a62153cef637"
server_key_version: "encryptionkey@0"
I20260924 16:12:41.345851 2574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:41.377668 2574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:41.381942 2574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:41.382213 2574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:41.383257 2574 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:12:41.384603 2574 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:12:41.384685 2574 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:41.384836 2574 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:12:41.384932 2574 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:41.419152 2574 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.2:34443
I20260924 16:12:41.419239 2702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.2:34443 every 8 connection(s)
I20260924 16:12:41.420023 2574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
I20260924 16:12:41.423863 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 2574
I20260924 16:12:41.424054 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/minicluster-data/ts-1/wal/instance
I20260924 16:12:41.424557 30632 external_mini_cluster.cc:1428] Setting key 0c56b2700f81c49c0ee4f4f8a702d075
Sep 24 16:12:41 dist-test-slave-v5j9 krb5kdc[2333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266361, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.2@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:12:41.441337 2570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:41.422557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.2:60063 (local address 127.29.234.62:34777)
0924 16:12:41.422759 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:41.422765 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:41.424046 (+ 1281us) server_negotiation.cc:407] Connection header received
0924 16:12:41.426158 (+ 2112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:41.426163 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:41.426249 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:41.426379 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:41.427835 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:41.428533 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:41.430097 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:41.430639 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:41.430778 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:41.434902 (+ 4124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:41.434907 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:41.434910 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:41.434942 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:41.437896 (+ 2954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:41.438659 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:41.438664 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:41.438668 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:41.438781 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:41.439300 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:41.439308 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:41.439313 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:41.439734 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:41.439911 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:41.440641 (+ 730us) server_negotiation.cc:299] Negotiation successful
0924 16:12:41.440937 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 16:12:41.442142 2705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:41.423133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:34777 (local address 127.29.234.2:60063)
0924 16:12:41.423871 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:41.423914 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:41.425882 (+ 1968us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:41.426599 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:41.426608 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:41.427119 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:41.427662 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:41.427684 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:41.428687 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:41.428697 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:41.429956 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:41.429968 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:41.430871 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:41.430878 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:41.431127 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:41.431939 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:41.431971 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:41.434681 (+ 2710us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:41.438037 (+ 3356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:41.438046 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:41.438083 (+ 37us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:41.438520 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:41.438901 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:41.438906 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:41.438910 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:41.439138 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:41.439871 (+ 733us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:41.439877 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:41.440230 (+ 353us) client_negotiation.cc:769] Sending connection context
0924 16:12:41.440466 (+ 236us) client_negotiation.cc:240] Negotiation successful
0924 16:12:41.440904 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":287,"threads_started":1}
WARNING: no policy specified for kudu/127.29.234.3@KRBTEST.COM; defaulting to no policy
I20260924 16:12:41.443774 2703 heartbeater.cc:347] Connected to a master server at 127.29.234.62:34777
I20260924 16:12:41.444149 2703 heartbeater.cc:464] Registering TS with master...
I20260924 16:12:41.444907 2703 heartbeater.cc:511] Master 127.29.234.62:34777 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.3@KRBTEST.COM" created.
I20260924 16:12:41.446768 2377 ts_manager.cc:194] Registered new tserver with Master: e3e1eeaa41064392900d85f3cde27aba (127.29.234.2:34443)
I20260924 16:12:41.448104 2377 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.234.2@KRBTEST.COM'} at 127.29.234.2:60063
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab.
Entry for principal kudu/127.29.234.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab.
I20260924 16:12:41.470563 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.3:0
--local_ip_for_outbound_sockets=127.29.234.3
--webserver_interface=127.29.234.3
--webserver_port=0
--tserver_master_addrs=127.29.234.62:34777
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab
--principal=kudu/127.29.234.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:45593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:12:41.794545 2708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:41.794914 2708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:41.795100 2708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:41.802313 2708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:41.802492 2708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:41.802686 2708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.3
I20260924 16:12:41.812624 2708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:45593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab
--principal=kudu/127.29.234.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.234.3
--webserver_port=0
--tserver_master_addrs=127.29.234.62:34777
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.3
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:12:41.814141 2708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:41.815500 2708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:41.828123 2713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:41.828167 2716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:41.828177 2714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:41.828970 2708 server_base.cc:1034] running on GCE node
I20260924 16:12:41.829610 2708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:41.830621 2708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:41.831825 2708 hybrid_clock.cc:648] HybridClock initialized: now 1790266361831772 us; error 49 us; skew 500 ppm
I20260924 16:12:41.832149 2708 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:12:41 dist-test-slave-v5j9 krb5kdc[2333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266361, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:41.835450 2708 init.cc:377] Logged in from keytab as kudu/127.29.234.3@KRBTEST.COM (short username kudu)
I20260924 16:12:41.836611 2708 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:41.836756 2708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:41.837074 2708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:41.840246 2708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/minicluster-data/ts-2/data/instance:
uuid: "5b7428b2404a459fa9e21911a6a872c5"
format_stamp: "Formatted at 2026-09-24 16:12:41 on dist-test-slave-v5j9"
server_key: "86089e01f3694ef03a1401c3293d7436"
server_key_iv: "da82cb592483b47bb63e43d3218d03b8"
server_key_version: "encryptionkey@0"
I20260924 16:12:41.841042 2708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/minicluster-data/ts-2/wal/instance:
uuid: "5b7428b2404a459fa9e21911a6a872c5"
format_stamp: "Formatted at 2026-09-24 16:12:41 on dist-test-slave-v5j9"
server_key: "86089e01f3694ef03a1401c3293d7436"
server_key_iv: "da82cb592483b47bb63e43d3218d03b8"
server_key_version: "encryptionkey@0"
I20260924 16:12:41.848300 2708 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.002s
I20260924 16:12:41.852041 2722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:41.853554 2708 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:12:41.853729 2708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/minicluster-data/ts-2/wal
uuid: "5b7428b2404a459fa9e21911a6a872c5"
format_stamp: "Formatted at 2026-09-24 16:12:41 on dist-test-slave-v5j9"
server_key: "86089e01f3694ef03a1401c3293d7436"
server_key_iv: "da82cb592483b47bb63e43d3218d03b8"
server_key_version: "encryptionkey@0"
I20260924 16:12:41.853952 2708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:41.876217 2708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:41.880092 2708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:41.880381 2708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:41.881423 2708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:12:41.882769 2708 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:12:41.882905 2708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:41.883036 2708 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:12:41.883126 2708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:41.916946 2708 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.3:42495
I20260924 16:12:41.916988 2836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.3:42495 every 8 connection(s)
I20260924 16:12:41.917871 2708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
I20260924 16:12:41.919484 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 2708
I20260924 16:12:41.919697 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/minicluster-data/ts-2/wal/instance
I20260924 16:12:41.920142 30632 external_mini_cluster.cc:1428] Setting key ac22b42bd94364da103e2be903175e1c
Sep 24 16:12:41 dist-test-slave-v5j9 krb5kdc[2333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266361, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.3@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:12:41.938462 2570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:41.920483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.3:56019 (local address 127.29.234.62:34777)
0924 16:12:41.920697 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:41.920704 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:41.921997 (+ 1293us) server_negotiation.cc:407] Connection header received
0924 16:12:41.924221 (+ 2224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:41.924227 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:41.924313 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:41.924434 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:41.925973 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:41.926718 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:41.928268 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:41.928702 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:41.928835 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:41.932403 (+ 3568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:41.932407 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:41.932411 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:41.932444 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:41.935449 (+ 3005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:41.936333 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:41.936337 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:41.936340 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:41.936464 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:41.936846 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:41.936850 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:41.936852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:41.937114 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:41.937243 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:41.938035 (+ 792us) server_negotiation.cc:299] Negotiation successful
0924 16:12:41.938222 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 16:12:41.939430 2839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:41.920976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:34777 (local address 127.29.234.3:56019)
0924 16:12:41.921842 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:41.921891 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:41.923959 (+ 2068us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:41.924665 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:41.924674 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:41.925177 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:41.925799 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:41.925816 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:41.926884 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:41.926888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:41.928122 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:41.928134 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:41.928946 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:41.928954 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:41.929169 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:41.929912 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:41.929934 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:41.932197 (+ 2263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:41.935692 (+ 3495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:41.935699 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:41.935712 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:41.936203 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:41.936557 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:41.936560 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:41.936562 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:41.936725 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:41.937249 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:41.937256 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:41.937671 (+ 415us) client_negotiation.cc:769] Sending connection context
0924 16:12:41.937925 (+ 254us) client_negotiation.cc:240] Negotiation successful
0924 16:12:41.938345 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":400,"threads_started":1}
I20260924 16:12:41.940907 2837 heartbeater.cc:347] Connected to a master server at 127.29.234.62:34777
I20260924 16:12:41.941218 2837 heartbeater.cc:464] Registering TS with master...
I20260924 16:12:41.941919 2837 heartbeater.cc:511] Master 127.29.234.62:34777 requested a full tablet report, sending...
I20260924 16:12:41.943364 2377 ts_manager.cc:194] Registered new tserver with Master: 5b7428b2404a459fa9e21911a6a872c5 (127.29.234.3:42495)
I20260924 16:12:41.944166 2377 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.234.3@KRBTEST.COM'} at 127.29.234.3:56019
I20260924 16:12:41.947486 2568 heartbeater.cc:503] Master 127.29.234.62:34777 was elected leader, sending a full tablet report...
I20260924 16:12:41.948357 30632 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:12:41.964548 2570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:41.953075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51008 (local address 127.29.234.62:34777)
0924 16:12:41.953221 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:41.953226 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:41.953793 (+ 567us) server_negotiation.cc:407] Connection header received
0924 16:12:41.953922 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:41.953925 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:41.953981 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:41.954056 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:41.955474 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:41.956139 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:41.957455 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:41.957879 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:41.957985 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:41.960172 (+ 2187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:41.960177 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:41.960180 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:41.960205 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:41.962532 (+ 2327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:41.963085 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:41.963088 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:41.963090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:41.963205 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:41.963495 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:41.963498 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:41.963499 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:41.963755 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:41.963934 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:41.964183 (+ 249us) server_negotiation.cc:299] Negotiation successful
0924 16:12:41.964334 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 16:12:41.968487 2377 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51008:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:12:41.975040 2377 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:12:42.014806 2852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:41.987430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56994 (local address 127.29.234.2:34443)
0924 16:12:42.002760 (+ 15330us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:42.002774 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:42.003882 (+ 1108us) server_negotiation.cc:407] Connection header received
0924 16:12:42.004044 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:42.004057 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:42.004440 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:42.004698 (+ 258us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:12:42.005896 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:42.008020 (+ 2124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:42.011751 (+ 3731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:42.013316 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:42.013466 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:42.013663 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:42.014292 (+ 629us) server_negotiation.cc:299] Negotiation successful
0924 16:12:42.014479 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15090,"thread_start_us":15357,"threads_started":1}
I20260924 16:12:42.014804 2853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:41.986914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.2:34443 (local address 127.0.0.1:56994)
0924 16:12:42.003512 (+ 16598us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:42.003538 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:42.003710 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:42.005059 (+ 1349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:42.005072 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:42.005114 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:12:42.005644 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:42.005661 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:42.008210 (+ 2549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:42.008219 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:42.011520 (+ 3301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:42.011535 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:42.013798 (+ 2263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:42.013805 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:42.014025 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:42.014096 (+ 71us) client_negotiation.cc:769] Sending connection context
0924 16:12:42.014220 (+ 124us) client_negotiation.cc:240] Negotiation successful
0924 16:12:42.014417 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16418,"thread_start_us":21519,"threads_started":1}
I20260924 16:12:42.016664 2851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:41.988200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:37255 (local address 127.0.0.1:43614)
0924 16:12:42.008721 (+ 20521us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:42.008744 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:42.008866 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:42.009931 (+ 1065us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:42.009937 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:42.009964 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:12:42.010472 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:42.010484 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:42.012224 (+ 1740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:42.012229 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:42.014116 (+ 1887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:42.014132 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:42.015890 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:42.015899 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:42.016065 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:42.016096 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 16:12:42.016189 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 16:12:42.016312 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20365,"thread_start_us":14976,"threads_started":1}
I20260924 16:12:42.016786 2854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:41.988583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43614 (local address 127.29.234.1:37255)
0924 16:12:42.006050 (+ 17467us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:42.006063 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:42.009007 (+ 2944us) server_negotiation.cc:407] Connection header received
0924 16:12:42.009123 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:42.009132 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:42.009456 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:42.009665 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:12:42.010679 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:42.012077 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:42.014313 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:42.015688 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:42.015832 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:42.016075 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:42.016263 (+ 188us) server_negotiation.cc:299] Negotiation successful
0924 16:12:42.016420 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17204,"thread_start_us":17467,"threads_started":1}
I20260924 16:12:42.017798 2849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:41.986556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.3:42495 (local address 127.0.0.1:48880)
0924 16:12:41.987815 (+ 1259us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:41.987864 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:41.988062 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:42.009663 (+ 21601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:42.009669 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:42.009700 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:12:42.010211 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:42.010227 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:42.011652 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:42.011660 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:42.013900 (+ 2240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:42.013913 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:42.016965 (+ 3052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:42.016971 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:42.017134 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:42.017166 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 16:12:42.017270 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 16:12:42.017441 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":920,"mutex_wait_us":60,"thread_start_us":748,"threads_started":1}
I20260924 16:12:42.019248 2638 tablet_service.cc:1467] Processing CreateTablet for tablet aa28484ddfe542baaf6aa6e0b69f569e (DEFAULT_TABLE table=test-table [id=5b14d81b61164215bbbd5f7de3b8cf95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:12:42.017988 2850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:41.986830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48880 (local address 127.29.234.3:42495)
0924 16:12:42.009152 (+ 22322us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:42.009161 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:42.009189 (+ 28us) server_negotiation.cc:407] Connection header received
0924 16:12:42.009266 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:42.009272 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:42.009479 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:42.009664 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:12:42.010431 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:42.011518 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:42.014073 (+ 2555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:42.015555 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:42.016594 (+ 1039us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:42.016830 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:42.017483 (+ 653us) server_negotiation.cc:299] Negotiation successful
0924 16:12:42.017648 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22143,"thread_start_us":21966,"threads_started":1}
I20260924 16:12:42.020170 2502 tablet_service.cc:1467] Processing CreateTablet for tablet aa28484ddfe542baaf6aa6e0b69f569e (DEFAULT_TABLE table=test-table [id=5b14d81b61164215bbbd5f7de3b8cf95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:12:42.020743 2638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa28484ddfe542baaf6aa6e0b69f569e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:42.021464 2502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa28484ddfe542baaf6aa6e0b69f569e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:42.023465 2771 tablet_service.cc:1467] Processing CreateTablet for tablet aa28484ddfe542baaf6aa6e0b69f569e (DEFAULT_TABLE table=test-table [id=5b14d81b61164215bbbd5f7de3b8cf95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:12:42.024864 2771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa28484ddfe542baaf6aa6e0b69f569e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:42.033018 2855 tablet_bootstrap.cc:492] T aa28484ddfe542baaf6aa6e0b69f569e P b77397a5eec341d2b526a5c6df0b520f: Bootstrap starting.
I20260924 16:12:42.036103 2855 tablet_bootstrap.cc:654] T aa28484ddfe542baaf6aa6e0b69f569e P b77397a5eec341d2b526a5c6df0b520f: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:42.036972 2856 tablet_bootstrap.cc:492] T aa28484ddfe542baaf6aa6e0b69f569e P e3e1eeaa41064392900d85f3cde27aba: Bootstrap starting.
I20260924 16:12:42.038236 2855 log.cc:824] T aa28484ddfe542baaf6aa6e0b69f569e P b77397a5eec341d2b526a5c6df0b520f: Log is configured to *not* fsync() on all Append() calls
I20260924 16:12:42.040692 2856 tablet_bootstrap.cc:654] T aa28484ddfe542baaf6aa6e0b69f569e P e3e1eeaa41064392900d85f3cde27aba: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:42.041378 2858 tablet_bootstrap.cc:492] T aa28484ddfe542baaf6aa6e0b69f569e P 5b7428b2404a459fa9e21911a6a872c5: Bootstrap starting.
I20260924 16:12:42.042047 2855 tablet_bootstrap.cc:492] T aa28484ddfe542baaf6aa6e0b69f569e P b77397a5eec341d2b526a5c6df0b520f: No bootstrap required, opened a new log
I20260924 16:12:42.042028 2856 log.cc:824] T aa28484ddfe542baaf6aa6e0b69f569e P e3e1eeaa41064392900d85f3cde27aba: Log is configured to *not* fsync() on all Append() calls
I20260924 16:12:42.042327 2855 ts_tablet_manager.cc:1397] T aa28484ddfe542baaf6aa6e0b69f569e P b77397a5eec341d2b526a5c6df0b520f: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260924 16:12:42.044710 2858 tablet_bootstrap.cc:654] T aa28484ddfe542baaf6aa6e0b69f569e P 5b7428b2404a459fa9e21911a6a872c5: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:42.045164 2856 tablet_bootstrap.cc:492] T aa28484ddfe542baaf6aa6e0b69f569e P e3e1eeaa41064392900d85f3cde27aba: No bootstrap required, opened a new log
I20260924 16:12:42.045483 2856 ts_tablet_manager.cc:1397] T aa28484ddfe542baaf6aa6e0b69f569e P e3e1eeaa41064392900d85f3cde27aba: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260924 16:12:42.045979 2858 log.cc:824] T aa28484ddfe542baaf6aa6e0b69f569e P 5b7428b2404a459fa9e21911a6a872c5: Log is configured to *not* fsync() on all Append() calls
I20260924 16:12:42.048802 2858 tablet_bootstrap.cc:492] T aa28484ddfe542baaf6aa6e0b69f569e P 5b7428b2404a459fa9e21911a6a872c5: No bootstrap required, opened a new log
I20260924 16:12:42.049062 2858 ts_tablet_manager.cc:1397] T aa28484ddfe542baaf6aa6e0b69f569e P 5b7428b2404a459fa9e21911a6a872c5: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260924 16:12:42.053068 2855 raft_consensus.cc:348] T aa28484ddfe542baaf6aa6e0b69f569e P b77397a5eec341d2b526a5c6df0b520f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7428b2404a459fa9e21911a6a872c5" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 42495 } } peers { permanent_uuid: "e3e1eeaa41064392900d85f3cde27aba" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 34443 } } peers { permanent_uuid: "b77397a5eec341d2b526a5c6df0b520f" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 37255 } }
I20260924 16:12:42.053544 2855 raft_consensus.cc:374] T aa28484ddfe542baaf6aa6e0b69f569e P b77397a5eec341d2b526a5c6df0b520f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:42.053681 2855 raft_consensus.cc:729] T aa28484ddfe542baaf6aa6e0b69f569e P b77397a5eec341d2b526a5c6df0b520f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b77397a5eec341d2b526a5c6df0b520f, State: Initialized, Role: FOLLOWER
I20260924 16:12:42.054350 2855 consensus_queue.cc:260] T aa28484ddfe542baaf6aa6e0b69f569e P b77397a5eec341d2b526a5c6df0b520f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7428b2404a459fa9e21911a6a872c5" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 42495 } } peers { permanent_uuid: "e3e1eeaa41064392900d85f3cde27aba" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 34443 } } peers { permanent_uuid: "b77397a5eec341d2b526a5c6df0b520f" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 37255 } }
I20260924 16:12:42.056218 2855 ts_tablet_manager.cc:1428] T aa28484ddfe542baaf6aa6e0b69f569e P b77397a5eec341d2b526a5c6df0b520f: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260924 16:12:42.056198 2856 raft_consensus.cc:348] T aa28484ddfe542baaf6aa6e0b69f569e P e3e1eeaa41064392900d85f3cde27aba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7428b2404a459fa9e21911a6a872c5" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 42495 } } peers { permanent_uuid: "e3e1eeaa41064392900d85f3cde27aba" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 34443 } } peers { permanent_uuid: "b77397a5eec341d2b526a5c6df0b520f" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 37255 } }
I20260924 16:12:42.056651 2856 raft_consensus.cc:374] T aa28484ddfe542baaf6aa6e0b69f569e P e3e1eeaa41064392900d85f3cde27aba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:42.056787 2856 raft_consensus.cc:729] T aa28484ddfe542baaf6aa6e0b69f569e P e3e1eeaa41064392900d85f3cde27aba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3e1eeaa41064392900d85f3cde27aba, State: Initialized, Role: FOLLOWER
I20260924 16:12:42.057485 2856 consensus_queue.cc:260] T aa28484ddfe542baaf6aa6e0b69f569e P e3e1eeaa41064392900d85f3cde27aba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7428b2404a459fa9e21911a6a872c5" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 42495 } } peers { permanent_uuid: "e3e1eeaa41064392900d85f3cde27aba" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 34443 } } peers { permanent_uuid: "b77397a5eec341d2b526a5c6df0b520f" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 37255 } }
I20260924 16:12:42.058344 2858 raft_consensus.cc:348] T aa28484ddfe542baaf6aa6e0b69f569e P 5b7428b2404a459fa9e21911a6a872c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7428b2404a459fa9e21911a6a872c5" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 42495 } } peers { permanent_uuid: "e3e1eeaa41064392900d85f3cde27aba" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 34443 } } peers { permanent_uuid: "b77397a5eec341d2b526a5c6df0b520f" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 37255 } }
I20260924 16:12:42.058840 2703 heartbeater.cc:503] Master 127.29.234.62:34777 was elected leader, sending a full tablet report...
I20260924 16:12:42.058851 2858 raft_consensus.cc:374] T aa28484ddfe542baaf6aa6e0b69f569e P 5b7428b2404a459fa9e21911a6a872c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:42.058949 2858 raft_consensus.cc:729] T aa28484ddfe542baaf6aa6e0b69f569e P 5b7428b2404a459fa9e21911a6a872c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b7428b2404a459fa9e21911a6a872c5, State: Initialized, Role: FOLLOWER
I20260924 16:12:42.059382 2856 ts_tablet_manager.cc:1428] T aa28484ddfe542baaf6aa6e0b69f569e P e3e1eeaa41064392900d85f3cde27aba: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260924 16:12:42.059566 2858 consensus_queue.cc:260] T aa28484ddfe542baaf6aa6e0b69f569e P 5b7428b2404a459fa9e21911a6a872c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7428b2404a459fa9e21911a6a872c5" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 42495 } } peers { permanent_uuid: "e3e1eeaa41064392900d85f3cde27aba" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 34443 } } peers { permanent_uuid: "b77397a5eec341d2b526a5c6df0b520f" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 37255 } }
I20260924 16:12:42.060787 2837 heartbeater.cc:503] Master 127.29.234.62:34777 was elected leader, sending a full tablet report...
I20260924 16:12:42.061401 2858 ts_tablet_manager.cc:1428] T aa28484ddfe542baaf6aa6e0b69f569e P 5b7428b2404a459fa9e21911a6a872c5: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260924 16:12:42.074090 2863 raft_consensus.cc:482] T aa28484ddfe542baaf6aa6e0b69f569e P 5b7428b2404a459fa9e21911a6a872c5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:12:42.074340 2863 raft_consensus.cc:504] T aa28484ddfe542baaf6aa6e0b69f569e P 5b7428b2404a459fa9e21911a6a872c5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7428b2404a459fa9e21911a6a872c5" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 42495 } } peers { permanent_uuid: "e3e1eeaa41064392900d85f3cde27aba" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 34443 } } peers { permanent_uuid: "b77397a5eec341d2b526a5c6df0b520f" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 37255 } }
I20260924 16:12:42.075850 2863 leader_election.cc:288] T aa28484ddfe542baaf6aa6e0b69f569e P 5b7428b2404a459fa9e21911a6a872c5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3e1eeaa41064392900d85f3cde27aba (127.29.234.2:34443), b77397a5eec341d2b526a5c6df0b520f (127.29.234.1:37255)
I20260924 16:12:42.082087 2839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:42.076026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.2:34443 (local address 127.29.234.3:47269)
0924 16:12:42.076350 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:42.076371 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:42.076509 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:42.076897 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:42.076902 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:42.076928 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:12:42.077142 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:42.077155 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:42.078361 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:42.078364 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:42.080151 (+ 1787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:42.080165 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:42.081490 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:42.081494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:42.081616 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:42.081638 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 16:12:42.081711 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 16:12:42.081799 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146}
I20260924 16:12:42.082127 2852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:42.076107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.3:47269 (local address 127.29.234.2:34443)
0924 16:12:42.076365 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:42.076372 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:42.076461 (+ 89us) server_negotiation.cc:407] Connection header received
0924 16:12:42.076675 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:42.076684 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:42.076766 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:42.076870 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:12:42.077270 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:42.078240 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:42.080288 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:42.081317 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:42.081493 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:42.081567 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:42.081766 (+ 199us) server_negotiation.cc:299] Negotiation successful
0924 16:12:42.081878 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260924 16:12:42.082799 2864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:42.076245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:37255 (local address 127.29.234.3:34623)
0924 16:12:42.076985 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:42.077006 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:42.077115 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:42.077455 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:42.077458 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:42.077477 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:12:42.077728 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:42.077734 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:42.078844 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:42.078851 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:42.080522 (+ 1671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:42.080531 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:42.082042 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:42.082047 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:42.082357 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:42.082398 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 16:12:42.082476 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 16:12:42.082596 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"thread_start_us":421,"threads_started":1}
I20260924 16:12:42.082849 2854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:42.076320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.3:34623 (local address 127.29.234.1:37255)
0924 16:12:42.076508 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:42.076513 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:42.077086 (+ 573us) server_negotiation.cc:407] Connection header received
0924 16:12:42.077240 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:42.077247 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:42.077319 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:42.077440 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:12:42.077891 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:42.078703 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:42.080643 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:42.081838 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:42.081994 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:42.082058 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:42.082508 (+ 450us) server_negotiation.cc:299] Negotiation successful
0924 16:12:42.082636 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 16:12:42.083032 2658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa28484ddfe542baaf6aa6e0b69f569e" candidate_uuid: "5b7428b2404a459fa9e21911a6a872c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3e1eeaa41064392900d85f3cde27aba" is_pre_election: true
I20260924 16:12:42.083519 2658 raft_consensus.cc:2454] T aa28484ddfe542baaf6aa6e0b69f569e P e3e1eeaa41064392900d85f3cde27aba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b7428b2404a459fa9e21911a6a872c5 in term 0.
I20260924 16:12:42.083698 2522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa28484ddfe542baaf6aa6e0b69f569e" candidate_uuid: "5b7428b2404a459fa9e21911a6a872c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b77397a5eec341d2b526a5c6df0b520f" is_pre_election: true
I20260924 16:12:42.084102 2724 leader_election.cc:302] T aa28484ddfe542baaf6aa6e0b69f569e P 5b7428b2404a459fa9e21911a6a872c5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b7428b2404a459fa9e21911a6a872c5, e3e1eeaa41064392900d85f3cde27aba; no voters:
I20260924 16:12:42.084177 2522 raft_consensus.cc:2454] T aa28484ddfe542baaf6aa6e0b69f569e P b77397a5eec341d2b526a5c6df0b520f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b7428b2404a459fa9e21911a6a872c5 in term 0.
I20260924 16:12:42.084483 2863 raft_consensus.cc:2781] T aa28484ddfe542baaf6aa6e0b69f569e P 5b7428b2404a459fa9e21911a6a872c5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:12:42.084622 2863 raft_consensus.cc:482] T aa28484ddfe542baaf6aa6e0b69f569e P 5b7428b2404a459fa9e21911a6a872c5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:12:42.084746 2863 raft_consensus.cc:3037] T aa28484ddfe542baaf6aa6e0b69f569e P 5b7428b2404a459fa9e21911a6a872c5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:42.087338 2863 raft_consensus.cc:504] T aa28484ddfe542baaf6aa6e0b69f569e P 5b7428b2404a459fa9e21911a6a872c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7428b2404a459fa9e21911a6a872c5" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 42495 } } peers { permanent_uuid: "e3e1eeaa41064392900d85f3cde27aba" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 34443 } } peers { permanent_uuid: "b77397a5eec341d2b526a5c6df0b520f" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 37255 } }
I20260924 16:12:42.087931 2863 leader_election.cc:288] T aa28484ddfe542baaf6aa6e0b69f569e P 5b7428b2404a459fa9e21911a6a872c5 [CANDIDATE]: Term 1 election: Requested vote from peers e3e1eeaa41064392900d85f3cde27aba (127.29.234.2:34443), b77397a5eec341d2b526a5c6df0b520f (127.29.234.1:37255)
I20260924 16:12:42.088263 2658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa28484ddfe542baaf6aa6e0b69f569e" candidate_uuid: "5b7428b2404a459fa9e21911a6a872c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3e1eeaa41064392900d85f3cde27aba"
I20260924 16:12:42.088423 2522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa28484ddfe542baaf6aa6e0b69f569e" candidate_uuid: "5b7428b2404a459fa9e21911a6a872c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b77397a5eec341d2b526a5c6df0b520f"
I20260924 16:12:42.088481 2658 raft_consensus.cc:3037] T aa28484ddfe542baaf6aa6e0b69f569e P e3e1eeaa41064392900d85f3cde27aba [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:42.088616 2522 raft_consensus.cc:3037] T aa28484ddfe542baaf6aa6e0b69f569e P b77397a5eec341d2b526a5c6df0b520f [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:42.091703 2658 raft_consensus.cc:2454] T aa28484ddfe542baaf6aa6e0b69f569e P e3e1eeaa41064392900d85f3cde27aba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b7428b2404a459fa9e21911a6a872c5 in term 1.
I20260924 16:12:42.091703 2522 raft_consensus.cc:2454] T aa28484ddfe542baaf6aa6e0b69f569e P b77397a5eec341d2b526a5c6df0b520f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b7428b2404a459fa9e21911a6a872c5 in term 1.
I20260924 16:12:42.092098 2724 leader_election.cc:302] T aa28484ddfe542baaf6aa6e0b69f569e P 5b7428b2404a459fa9e21911a6a872c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b7428b2404a459fa9e21911a6a872c5, e3e1eeaa41064392900d85f3cde27aba; no voters:
I20260924 16:12:42.092355 2863 raft_consensus.cc:2781] T aa28484ddfe542baaf6aa6e0b69f569e P 5b7428b2404a459fa9e21911a6a872c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:12:42.092746 2863 raft_consensus.cc:686] T aa28484ddfe542baaf6aa6e0b69f569e P 5b7428b2404a459fa9e21911a6a872c5 [term 1 LEADER]: Becoming Leader. State: Replica: 5b7428b2404a459fa9e21911a6a872c5, State: Running, Role: LEADER
I20260924 16:12:42.093205 2863 consensus_queue.cc:237] T aa28484ddfe542baaf6aa6e0b69f569e P 5b7428b2404a459fa9e21911a6a872c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7428b2404a459fa9e21911a6a872c5" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 42495 } } peers { permanent_uuid: "e3e1eeaa41064392900d85f3cde27aba" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 34443 } } peers { permanent_uuid: "b77397a5eec341d2b526a5c6df0b520f" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 37255 } }
I20260924 16:12:42.097885 2377 catalog_manager.cc:5495] T aa28484ddfe542baaf6aa6e0b69f569e P 5b7428b2404a459fa9e21911a6a872c5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b7428b2404a459fa9e21911a6a872c5 (127.29.234.3). New cstate: current_term: 1 leader_uuid: "5b7428b2404a459fa9e21911a6a872c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7428b2404a459fa9e21911a6a872c5" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 42495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3e1eeaa41064392900d85f3cde27aba" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 34443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b77397a5eec341d2b526a5c6df0b520f" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 37255 } health_report { overall_health: UNKNOWN } } }
I20260924 16:12:42.121299 2850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:42.116432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48890 (local address 127.29.234.3:42495)
0924 16:12:42.116600 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:42.116605 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:42.116762 (+ 157us) server_negotiation.cc:407] Connection header received
0924 16:12:42.116903 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:42.116906 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:42.116978 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:42.117094 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:12:42.117528 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:42.118220 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:42.119455 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:42.119833 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:42.119920 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:42.120338 (+ 418us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:12:42.120855 (+ 517us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:12:42.120976 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:42.121038 (+ 62us) server_negotiation.cc:299] Negotiation successful
0924 16:12:42.121099 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 16:12:42.129472 2658 raft_consensus.cc:1261] T aa28484ddfe542baaf6aa6e0b69f569e P e3e1eeaa41064392900d85f3cde27aba [term 1 FOLLOWER]: Refusing update from remote peer 5b7428b2404a459fa9e21911a6a872c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:12:42.129472 2522 raft_consensus.cc:1261] T aa28484ddfe542baaf6aa6e0b69f569e P b77397a5eec341d2b526a5c6df0b520f [term 1 FOLLOWER]: Refusing update from remote peer 5b7428b2404a459fa9e21911a6a872c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:12:42.130630 2865 consensus_queue.cc:1035] T aa28484ddfe542baaf6aa6e0b69f569e P 5b7428b2404a459fa9e21911a6a872c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b77397a5eec341d2b526a5c6df0b520f" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 37255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:12:42.131006 2863 consensus_queue.cc:1035] T aa28484ddfe542baaf6aa6e0b69f569e P 5b7428b2404a459fa9e21911a6a872c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3e1eeaa41064392900d85f3cde27aba" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 34443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:12:42.142046 2868 mvcc.cc:205] Tried to move back new op lower bound from 7332931019271757824 to 7332931019138961408. Current Snapshot: MvccSnapshot[applied={T|T < 7332931019271757824}]
I20260924 16:12:42.146487 2869 mvcc.cc:205] Tried to move back new op lower bound from 7332931019271757824 to 7332931019138961408. Current Snapshot: MvccSnapshot[applied={T|T < 7332931019271757824}]
I20260924 16:12:42.158036 2376 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51008:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:12:42.158380 2376 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51008:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:12:42.162078 2376 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 66c70841ea7d4421b09018f49633fb95: Sending DeleteTablet for 3 replicas of tablet aa28484ddfe542baaf6aa6e0b69f569e
I20260924 16:12:42.163218 2771 tablet_service.cc:1514] Processing DeleteTablet for tablet aa28484ddfe542baaf6aa6e0b69f569e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:12:42 UTC) from {username='kudu', principal='kudu/127.29.234.62@KRBTEST.COM'} at 127.0.0.1:48880
I20260924 16:12:42.163287 2638 tablet_service.cc:1514] Processing DeleteTablet for tablet aa28484ddfe542baaf6aa6e0b69f569e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:12:42 UTC) from {username='kudu', principal='kudu/127.29.234.62@KRBTEST.COM'} at 127.0.0.1:56994
I20260924 16:12:42.163878 2502 tablet_service.cc:1514] Processing DeleteTablet for tablet aa28484ddfe542baaf6aa6e0b69f569e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:12:42 UTC) from {username='kudu', principal='kudu/127.29.234.62@KRBTEST.COM'} at 127.0.0.1:43614
I20260924 16:12:42.164386 2878 tablet_replica.cc:330] T aa28484ddfe542baaf6aa6e0b69f569e P e3e1eeaa41064392900d85f3cde27aba: stopping tablet replica
I20260924 16:12:42.165143 2878 raft_consensus.cc:2229] T aa28484ddfe542baaf6aa6e0b69f569e P e3e1eeaa41064392900d85f3cde27aba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:12:42.165750 2878 raft_consensus.cc:2258] T aa28484ddfe542baaf6aa6e0b69f569e P e3e1eeaa41064392900d85f3cde27aba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:12:42.166500 2877 tablet_replica.cc:330] T aa28484ddfe542baaf6aa6e0b69f569e P 5b7428b2404a459fa9e21911a6a872c5: stopping tablet replica
I20260924 16:12:42.166589 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 2440
I20260924 16:12:42.166899 2877 raft_consensus.cc:2229] T aa28484ddfe542baaf6aa6e0b69f569e P 5b7428b2404a459fa9e21911a6a872c5 [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:12:42.167496 2877 raft_consensus.cc:2258] T aa28484ddfe542baaf6aa6e0b69f569e P 5b7428b2404a459fa9e21911a6a872c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:12:42.168560 2878 ts_tablet_manager.cc:1905] T aa28484ddfe542baaf6aa6e0b69f569e P e3e1eeaa41064392900d85f3cde27aba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:12:42.170457 2879 tablet_replica.cc:330] T aa28484ddfe542baaf6aa6e0b69f569e P b77397a5eec341d2b526a5c6df0b520f: stopping tablet replica
I20260924 16:12:42.170886 2879 raft_consensus.cc:2229] T aa28484ddfe542baaf6aa6e0b69f569e P b77397a5eec341d2b526a5c6df0b520f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:12:42.170966 2877 ts_tablet_manager.cc:1905] T aa28484ddfe542baaf6aa6e0b69f569e P 5b7428b2404a459fa9e21911a6a872c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:12:42.171375 2879 raft_consensus.cc:2258] T aa28484ddfe542baaf6aa6e0b69f569e P b77397a5eec341d2b526a5c6df0b520f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:12:42.173223 2879 ts_tablet_manager.cc:1905] T aa28484ddfe542baaf6aa6e0b69f569e P b77397a5eec341d2b526a5c6df0b520f: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 24 16:12:42 dist-test-slave-v5j9 krb5kdc[2333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.234.1@KRBTEST.COM
I20260924 16:12:42.180188 2879 ts_tablet_manager.cc:1918] T aa28484ddfe542baaf6aa6e0b69f569e P b77397a5eec341d2b526a5c6df0b520f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:12:42.180379 2879 log.cc:1196] T aa28484ddfe542baaf6aa6e0b69f569e P b77397a5eec341d2b526a5c6df0b520f: Deleting WAL directory at /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/minicluster-data/ts-0/wal/wals/aa28484ddfe542baaf6aa6e0b69f569e
I20260924 16:12:42.180933 2879 ts_tablet_manager.cc:1939] T aa28484ddfe542baaf6aa6e0b69f569e P b77397a5eec341d2b526a5c6df0b520f: Deleting consensus metadata
I20260924 16:12:42.180924 2878 ts_tablet_manager.cc:1918] T aa28484ddfe542baaf6aa6e0b69f569e P e3e1eeaa41064392900d85f3cde27aba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:12:42.181084 2878 log.cc:1196] T aa28484ddfe542baaf6aa6e0b69f569e P e3e1eeaa41064392900d85f3cde27aba: Deleting WAL directory at /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/minicluster-data/ts-1/wal/wals/aa28484ddfe542baaf6aa6e0b69f569e
I20260924 16:12:42.182963 2364 catalog_manager.cc:4841] TS b77397a5eec341d2b526a5c6df0b520f (127.29.234.1:37255): tablet aa28484ddfe542baaf6aa6e0b69f569e (table test-table [id=5b14d81b61164215bbbd5f7de3b8cf95]) successfully deleted
I20260924 16:12:42.183877 2878 ts_tablet_manager.cc:1939] T aa28484ddfe542baaf6aa6e0b69f569e P e3e1eeaa41064392900d85f3cde27aba: Deleting consensus metadata
I20260924 16:12:42.185612 2362 catalog_manager.cc:4841] TS e3e1eeaa41064392900d85f3cde27aba (127.29.234.2:34443): tablet aa28484ddfe542baaf6aa6e0b69f569e (table test-table [id=5b14d81b61164215bbbd5f7de3b8cf95]) successfully deleted
I20260924 16:12:42.185854 2877 ts_tablet_manager.cc:1918] T aa28484ddfe542baaf6aa6e0b69f569e P 5b7428b2404a459fa9e21911a6a872c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:12:42.186049 2877 log.cc:1196] T aa28484ddfe542baaf6aa6e0b69f569e P 5b7428b2404a459fa9e21911a6a872c5: Deleting WAL directory at /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790266323169505-30632-0/minicluster-data/ts-2/wal/wals/aa28484ddfe542baaf6aa6e0b69f569e
I20260924 16:12:42.186599 2877 ts_tablet_manager.cc:1939] T aa28484ddfe542baaf6aa6e0b69f569e P 5b7428b2404a459fa9e21911a6a872c5: Deleting consensus metadata
I20260924 16:12:42.187911 2364 catalog_manager.cc:4841] TS 5b7428b2404a459fa9e21911a6a872c5 (127.29.234.3:42495): tablet aa28484ddfe542baaf6aa6e0b69f569e (table test-table [id=5b14d81b61164215bbbd5f7de3b8cf95]) successfully deleted
I20260924 16:12:42.188580 2854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:42.168158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43624 (local address 127.29.234.1:37255)
0924 16:12:42.168383 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:42.168393 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:42.169860 (+ 1467us) server_negotiation.cc:407] Connection header received
0924 16:12:42.170032 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:42.170041 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:42.170156 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:42.170264 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:42.172066 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:42.173263 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:42.175151 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:42.175812 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:42.175923 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:42.181148 (+ 5225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:42.181162 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:42.181181 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:42.181227 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:42.185097 (+ 3870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:42.186163 (+ 1066us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:42.186174 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:42.186181 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:42.186333 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:42.186790 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:42.186795 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:42.186799 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:42.187253 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:42.187555 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:42.188135 (+ 580us) server_negotiation.cc:299] Negotiation successful
0924 16:12:42.188323 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 16:12:42.189275 2563 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:12:42.299256 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 2440
I20260924 16:12:42.322988 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 2574
Sep 24 16:12:42 dist-test-slave-v5j9 krb5kdc[2333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.234.2@KRBTEST.COM
I20260924 16:12:42.340233 2852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:42.324377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56996 (local address 127.29.234.2:34443)
0924 16:12:42.324580 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:42.324587 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:42.324607 (+ 20us) server_negotiation.cc:407] Connection header received
0924 16:12:42.324671 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:42.324698 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:42.324836 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:42.324945 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:42.326538 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:42.327256 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:42.329071 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:42.329619 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:42.329712 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:42.334642 (+ 4930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:42.334656 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:42.334674 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:42.334705 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:42.337579 (+ 2874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:42.338333 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:42.338340 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:42.338342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:42.338457 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:42.338835 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:42.338838 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:42.338840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:42.339195 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:42.339408 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:42.339749 (+ 341us) server_negotiation.cc:299] Negotiation successful
0924 16:12:42.339938 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 16:12:42.340952 2698 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:12:42.431895 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 2574
I20260924 16:12:42.450254 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 2708
Sep 24 16:12:42 dist-test-slave-v5j9 krb5kdc[2333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.234.3@KRBTEST.COM
I20260924 16:12:42.466971 2850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:42.451097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33692 (local address 127.29.234.3:42495)
0924 16:12:42.451275 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:42.451280 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:42.451468 (+ 188us) server_negotiation.cc:407] Connection header received
0924 16:12:42.451677 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:42.451683 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:42.451768 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:42.451878 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:42.453170 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:42.454109 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:42.455721 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:42.456285 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:42.456419 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:42.460914 (+ 4495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:42.460926 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:42.460942 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:42.460978 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:42.464064 (+ 3086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:42.464700 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:42.464714 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:42.464717 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:42.464867 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:42.465251 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:42.465270 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:42.465272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:42.465755 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:42.466100 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:42.466394 (+ 294us) server_negotiation.cc:299] Negotiation successful
0924 16:12:42.466617 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 16:12:42.467690 2832 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:12:42.588248 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 2708
I20260924 16:12:42.604588 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 2346
I20260924 16:12:42.605820 2408 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:12:42.714727 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 2346
2026-09-24T16:12:42Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4150 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-24T16:12:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:12:42Z Disabled control of system clock
I20260924 16:12:42.780737 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266323169505-30632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266323169505-30632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266323169505-30632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:42951
--webserver_interface=127.29.234.62
--webserver_port=0
--builtin_ntp_servers=127.29.234.20:37549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.234.62:42951
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 16:12:43.101457 2889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:43.101742 2889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:43.101826 2889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:43.109025 2889 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:12:43.109140 2889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:43.109186 2889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:43.109236 2889 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:12:43.109273 2889 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:12:43.118685 2889 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:37549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266323169505-30632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266323169505-30632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.234.62:42951
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:42951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.234.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266323169505-30632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:12:43.119963 2889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:43.121732 2889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:43.133849 2895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:43.133792 2896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:43.133827 2898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:43.134814 2889 server_base.cc:1034] running on GCE node
I20260924 16:12:43.135573 2889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:43.136960 2889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:43.138228 2889 hybrid_clock.cc:648] HybridClock initialized: now 1790266363138175 us; error 70 us; skew 500 ppm
I20260924 16:12:43.138576 2889 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 16:12:43.140789 2889 webserver.cc:460] Webserver started at http://127.29.234.62:45511/ using document root <none> and password file <none>
I20260924 16:12:43.141539 2889 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:43.141674 2889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:43.141996 2889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:43.144784 2889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266323169505-30632-0/minicluster-data/master-0/data/instance:
uuid: "3e1e4efebe5c4938b6a0517f7afd8820"
format_stamp: "Formatted at 2026-09-24 16:12:43 on dist-test-slave-v5j9"
server_key: "97ac6dc4f6f5340a26d977fe3fc6aecd"
server_key_iv: "1ba3f33a6a5184a8df7c3afbabbef55f"
server_key_version: "encryptionkey@0"
I20260924 16:12:43.145623 2889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266323169505-30632-0/minicluster-data/master-0/wal/instance:
uuid: "3e1e4efebe5c4938b6a0517f7afd8820"
format_stamp: "Formatted at 2026-09-24 16:12:43 on dist-test-slave-v5j9"
server_key: "97ac6dc4f6f5340a26d977fe3fc6aecd"
server_key_iv: "1ba3f33a6a5184a8df7c3afbabbef55f"
server_key_version: "encryptionkey@0"
I20260924 16:12:43.150506 2889 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 16:12:43.153779 2904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:43.154969 2889 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 16:12:43.155140 2889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266323169505-30632-0/minicluster-data/master-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266323169505-30632-0/minicluster-data/master-0/wal
uuid: "3e1e4efebe5c4938b6a0517f7afd8820"
format_stamp: "Formatted at 2026-09-24 16:12:43 on dist-test-slave-v5j9"
server_key: "97ac6dc4f6f5340a26d977fe3fc6aecd"
server_key_iv: "1ba3f33a6a5184a8df7c3afbabbef55f"
server_key_version: "encryptionkey@0"
I20260924 16:12:43.155352 2889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266323169505-30632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266323169505-30632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266323169505-30632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:43.160794 2889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:43.161604 2889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:43.161899 2889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:43.179735 2889 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.62:42951
I20260924 16:12:43.179774 2956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.62:42951 every 8 connection(s)
I20260924 16:12:43.181017 2889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
I20260924 16:12:43.184598 2957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:43.187626 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 2889
I20260924 16:12:43.187821 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266323169505-30632-0/minicluster-data/master-0/wal/instance
I20260924 16:12:43.188658 30632 external_mini_cluster.cc:1428] Setting key bd8647eedcdf1e200cf35dd415ec84e7
I20260924 16:12:43.197602 2960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:43.190894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46470 (local address 127.29.234.62:42951)
0924 16:12:43.192037 (+ 1143us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:43.192048 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:43.192091 (+ 43us) server_negotiation.cc:407] Connection header received
0924 16:12:43.193892 (+ 1801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:43.193915 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:43.194439 (+ 524us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:43.194791 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:43.195911 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:43.195918 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:43.195930 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:12:43.195944 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:43.196005 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:12:43.196029 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:43.196231 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:43.196330 (+ 99us) server_negotiation.cc:299] Negotiation successful
0924 16:12:43.196602 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":652,"thread_start_us":369,"threads_started":1}
I20260924 16:12:43.198055 2957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e1e4efebe5c4938b6a0517f7afd8820: Bootstrap starting.
I20260924 16:12:43.201107 2957 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e1e4efebe5c4938b6a0517f7afd8820: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:43.202210 2957 log.cc:824] T 00000000000000000000000000000000 P 3e1e4efebe5c4938b6a0517f7afd8820: Log is configured to *not* fsync() on all Append() calls
I20260924 16:12:43.204962 2957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e1e4efebe5c4938b6a0517f7afd8820: No bootstrap required, opened a new log
I20260924 16:12:43.212189 2957 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e1e4efebe5c4938b6a0517f7afd8820 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e1e4efebe5c4938b6a0517f7afd8820" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 42951 } }
I20260924 16:12:43.212613 2957 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e1e4efebe5c4938b6a0517f7afd8820 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:43.212723 2957 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e1e4efebe5c4938b6a0517f7afd8820 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e1e4efebe5c4938b6a0517f7afd8820, State: Initialized, Role: FOLLOWER
I20260924 16:12:43.213330 2957 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e1e4efebe5c4938b6a0517f7afd8820 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e1e4efebe5c4938b6a0517f7afd8820" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 42951 } }
I20260924 16:12:43.213632 2957 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e1e4efebe5c4938b6a0517f7afd8820 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:12:43.213790 2957 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e1e4efebe5c4938b6a0517f7afd8820 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:12:43.213963 2957 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e1e4efebe5c4938b6a0517f7afd8820 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:43.216141 2957 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e1e4efebe5c4938b6a0517f7afd8820 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e1e4efebe5c4938b6a0517f7afd8820" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 42951 } }
I20260924 16:12:43.216662 2957 leader_election.cc:302] T 00000000000000000000000000000000 P 3e1e4efebe5c4938b6a0517f7afd8820 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e1e4efebe5c4938b6a0517f7afd8820; no voters:
I20260924 16:12:43.217273 2957 leader_election.cc:288] T 00000000000000000000000000000000 P 3e1e4efebe5c4938b6a0517f7afd8820 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:12:43.217444 2962 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e1e4efebe5c4938b6a0517f7afd8820 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:12:43.218150 2962 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e1e4efebe5c4938b6a0517f7afd8820 [term 1 LEADER]: Becoming Leader. State: Replica: 3e1e4efebe5c4938b6a0517f7afd8820, State: Running, Role: LEADER
I20260924 16:12:43.218662 2962 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e1e4efebe5c4938b6a0517f7afd8820 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e1e4efebe5c4938b6a0517f7afd8820" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 42951 } }
I20260924 16:12:43.219467 2957 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e1e4efebe5c4938b6a0517f7afd8820 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:12:43.221629 2963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e1e4efebe5c4938b6a0517f7afd8820 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e1e4efebe5c4938b6a0517f7afd8820" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e1e4efebe5c4938b6a0517f7afd8820" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 42951 } } }
I20260924 16:12:43.222437 2963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e1e4efebe5c4938b6a0517f7afd8820 [sys.catalog]: This master's current role is: LEADER
I20260924 16:12:43.221606 2964 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e1e4efebe5c4938b6a0517f7afd8820 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e1e4efebe5c4938b6a0517f7afd8820. Latest consensus state: current_term: 1 leader_uuid: "3e1e4efebe5c4938b6a0517f7afd8820" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e1e4efebe5c4938b6a0517f7afd8820" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 42951 } } }
I20260924 16:12:43.222599 2964 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e1e4efebe5c4938b6a0517f7afd8820 [sys.catalog]: This master's current role is: LEADER
I20260924 16:12:43.224126 2969 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:12:43.229537 2969 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:12:43.238339 2969 catalog_manager.cc:1347] Generated new cluster ID: c0d216f4eaee4a8dbd2894bd9815a4b3
I20260924 16:12:43.238459 2969 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:12:43.257921 2969 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:12:43.258365 2969 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:12:43.267012 2969 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e1e4efebe5c4938b6a0517f7afd8820: Generated new TSK 0
I20260924 16:12:43.267799 2969 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:12:43.282312 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266323169505-30632-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266323169505-30632-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266323169505-30632-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.1:0
--local_ip_for_outbound_sockets=127.29.234.1
--webserver_interface=127.29.234.1
--webserver_port=0
--tserver_master_addrs=127.29.234.62:42951
--builtin_ntp_servers=127.29.234.20:37549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 16:12:43.616154 2981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:43.616443 2981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:43.616592 2981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:43.623986 2981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:43.624095 2981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:43.624267 2981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.1
I20260924 16:12:43.633455 2981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:37549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266323169505-30632-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266323169505-30632-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.234.1
--webserver_port=0
--tserver_master_addrs=127.29.234.62:42951
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.1
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266323169505-30632-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:12:43.634773 2981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:43.636523 2981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:43.649359 2986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:43.649299 2989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:43.649400 2987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:43.650241 2981 server_base.cc:1034] running on GCE node
I20260924 16:12:43.650808 2981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:43.651659 2981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:43.652881 2981 hybrid_clock.cc:648] HybridClock initialized: now 1790266363652849 us; error 53 us; skew 500 ppm
I20260924 16:12:43.653188 2981 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 16:12:43.655588 2981 webserver.cc:460] Webserver started at http://127.29.234.1:33555/ using document root <none> and password file <none>
I20260924 16:12:43.656294 2981 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:43.656430 2981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:43.656754 2981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:43.659776 2981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266323169505-30632-0/minicluster-data/ts-0/data/instance:
uuid: "0dc415726f1d4d329d599ae69300b95a"
format_stamp: "Formatted at 2026-09-24 16:12:43 on dist-test-slave-v5j9"
server_key: "7accce64495436529650f6ce421942bf"
server_key_iv: "d4e803c607ab1c6eac96776feadb479c"
server_key_version: "encryptionkey@0"
I20260924 16:12:43.660609 2981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266323169505-30632-0/minicluster-data/ts-0/wal/instance:
uuid: "0dc415726f1d4d329d599ae69300b95a"
format_stamp: "Formatted at 2026-09-24 16:12:43 on dist-test-slave-v5j9"
server_key: "7accce64495436529650f6ce421942bf"
server_key_iv: "d4e803c607ab1c6eac96776feadb479c"
server_key_version: "encryptionkey@0"
I20260924 16:12:43.665871 2981 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260924 16:12:43.669145 2995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:43.670491 2981 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:12:43.670670 2981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266323169505-30632-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266323169505-30632-0/minicluster-data/ts-0/wal
uuid: "0dc415726f1d4d329d599ae69300b95a"
format_stamp: "Formatted at 2026-09-24 16:12:43 on dist-test-slave-v5j9"
server_key: "7accce64495436529650f6ce421942bf"
server_key_iv: "d4e803c607ab1c6eac96776feadb479c"
server_key_version: "encryptionkey@0"
I20260924 16:12:43.670914 2981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266323169505-30632-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266323169505-30632-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266323169505-30632-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:43.676621 2981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:43.677421 2981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:43.677688 2981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:43.678566 2981 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:12:43.679903 2981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:12:43.680030 2981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:43.680150 2981 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:12:43.680234 2981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:43.712237 2981 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.1:34175
I20260924 16:12:43.712291 3110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.1:34175 every 8 connection(s)
I20260924 16:12:43.713670 2981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
I20260924 16:12:43.714370 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 2981
I20260924 16:12:43.714596 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266323169505-30632-0/minicluster-data/ts-0/wal/instance
I20260924 16:12:43.715396 30632 external_mini_cluster.cc:1428] Setting key 50e6e44e637e1c78bc7adce468336895
I20260924 16:12:43.718597 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266323169505-30632-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266323169505-30632-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266323169505-30632-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.2:0
--local_ip_for_outbound_sockets=127.29.234.2
--webserver_interface=127.29.234.2
--webserver_port=0
--tserver_master_addrs=127.29.234.62:42951
--builtin_ntp_servers=127.29.234.20:37549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260924 16:12:43.724622 3113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:43.716600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.1:53135 (local address 127.29.234.62:42951)
0924 16:12:43.717059 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:43.717066 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:43.718742 (+ 1676us) server_negotiation.cc:407] Connection header received
0924 16:12:43.721618 (+ 2876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:43.721625 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:43.721696 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:43.721814 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:43.723472 (+ 1658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:43.723481 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:43.723488 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:12:43.723521 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:43.723577 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:12:43.723598 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:43.723716 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:43.724058 (+ 342us) server_negotiation.cc:299] Negotiation successful
0924 16:12:43.724126 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":237,"threads_started":1}
I20260924 16:12:43.725739 3114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:43.717349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:42951 (local address 127.29.234.1:53135)
0924 16:12:43.718551 (+ 1202us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:43.718601 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:43.721266 (+ 2665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:43.722077 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:43.722091 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:43.722226 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:43.723039 (+ 813us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:12:43.723045 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:43.723098 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:12:43.723258 (+ 160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:43.723805 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:43.723819 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:43.723830 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 16:12:43.724108 (+ 278us) client_negotiation.cc:240] Negotiation successful
0924 16:12:43.724433 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":308,"threads_started":1}
I20260924 16:12:43.730746 3111 heartbeater.cc:347] Connected to a master server at 127.29.234.62:42951
I20260924 16:12:43.731122 3111 heartbeater.cc:464] Registering TS with master...
I20260924 16:12:43.731922 3111 heartbeater.cc:511] Master 127.29.234.62:42951 requested a full tablet report, sending...
I20260924 16:12:43.734863 2922 ts_manager.cc:194] Registered new tserver with Master: 0dc415726f1d4d329d599ae69300b95a (127.29.234.1:34175)
W20260924 16:12:44.059963 3115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:44.060236 3115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:44.060397 3115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:44.067595 3115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:44.067678 3115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:44.067816 3115 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.2
I20260924 16:12:44.076714 3115 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:37549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266323169505-30632-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266323169505-30632-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.234.2
--webserver_port=0
--tserver_master_addrs=127.29.234.62:42951
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.2
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266323169505-30632-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:12:44.077965 3115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:44.079725 3115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:44.092320 3120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:44.092298 3121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:44.092782 3123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:44.093142 3115 server_base.cc:1034] running on GCE node
I20260924 16:12:44.093807 3115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:44.094712 3115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:44.096051 3115 hybrid_clock.cc:648] HybridClock initialized: now 1790266364096014 us; error 44 us; skew 500 ppm
I20260924 16:12:44.096401 3115 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 16:12:44.098673 3115 webserver.cc:460] Webserver started at http://127.29.234.2:34005/ using document root <none> and password file <none>
I20260924 16:12:44.099376 3115 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:44.099510 3115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:44.099862 3115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:44.102779 3115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266323169505-30632-0/minicluster-data/ts-1/data/instance:
uuid: "c2696379a76c4f448e58c70cb3044f70"
format_stamp: "Formatted at 2026-09-24 16:12:44 on dist-test-slave-v5j9"
server_key: "189fe8a2379322cc3aadb8e34630cbe8"
server_key_iv: "4f5c8daf392efaf022c3d032c583e52e"
server_key_version: "encryptionkey@0"
I20260924 16:12:44.103643 3115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266323169505-30632-0/minicluster-data/ts-1/wal/instance:
uuid: "c2696379a76c4f448e58c70cb3044f70"
format_stamp: "Formatted at 2026-09-24 16:12:44 on dist-test-slave-v5j9"
server_key: "189fe8a2379322cc3aadb8e34630cbe8"
server_key_iv: "4f5c8daf392efaf022c3d032c583e52e"
server_key_version: "encryptionkey@0"
I20260924 16:12:44.108685 3115 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 16:12:44.112102 3129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:44.113337 3115 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 16:12:44.113554 3115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266323169505-30632-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266323169505-30632-0/minicluster-data/ts-1/wal
uuid: "c2696379a76c4f448e58c70cb3044f70"
format_stamp: "Formatted at 2026-09-24 16:12:44 on dist-test-slave-v5j9"
server_key: "189fe8a2379322cc3aadb8e34630cbe8"
server_key_iv: "4f5c8daf392efaf022c3d032c583e52e"
server_key_version: "encryptionkey@0"
I20260924 16:12:44.113775 3115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266323169505-30632-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266323169505-30632-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266323169505-30632-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:44.119141 3115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:44.119928 3115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:44.120189 3115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:44.121117 3115 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:12:44.122506 3115 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:12:44.122638 3115 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:44.122784 3115 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:12:44.122872 3115 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:44.154975 3115 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.2:41553
I20260924 16:12:44.155025 3246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.2:41553 every 8 connection(s)
I20260924 16:12:44.156344 3115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
I20260924 16:12:44.164745 3113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:44.158993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.2:42183 (local address 127.29.234.62:42951)
0924 16:12:44.159216 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:44.159220 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:44.160709 (+ 1489us) server_negotiation.cc:407] Connection header received
0924 16:12:44.162469 (+ 1760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:44.162474 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:44.162539 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:44.162659 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:44.163993 (+ 1334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:44.163998 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:44.164003 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:12:44.164021 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:44.164061 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:12:44.164072 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:44.164190 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:44.164412 (+ 222us) server_negotiation.cc:299] Negotiation successful
0924 16:12:44.164480 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260924 16:12:44.165763 3249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:44.159589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:42951 (local address 127.29.234.2:42183)
0924 16:12:44.160556 (+ 967us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:44.160591 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:44.162183 (+ 1592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:44.162885 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:44.162893 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:44.162981 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:44.163715 (+ 734us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:12:44.163718 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:44.163756 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:12:44.163834 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:44.164161 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:44.164171 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:44.164179 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 16:12:44.164495 (+ 316us) client_negotiation.cc:240] Negotiation successful
0924 16:12:44.164751 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":314,"threads_started":1}
I20260924 16:12:44.166152 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 3115
I20260924 16:12:44.166348 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266323169505-30632-0/minicluster-data/ts-1/wal/instance
I20260924 16:12:44.166805 30632 external_mini_cluster.cc:1428] Setting key 32b5c2881db908e6108792c96c1ae1c2
I20260924 16:12:44.167182 3247 heartbeater.cc:347] Connected to a master server at 127.29.234.62:42951
I20260924 16:12:44.167505 3247 heartbeater.cc:464] Registering TS with master...
I20260924 16:12:44.168072 3247 heartbeater.cc:511] Master 127.29.234.62:42951 requested a full tablet report, sending...
I20260924 16:12:44.169612 2922 ts_manager.cc:194] Registered new tserver with Master: c2696379a76c4f448e58c70cb3044f70 (127.29.234.2:41553)
I20260924 16:12:44.169726 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266323169505-30632-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266323169505-30632-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266323169505-30632-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.3:0
--local_ip_for_outbound_sockets=127.29.234.3
--webserver_interface=127.29.234.3
--webserver_port=0
--tserver_master_addrs=127.29.234.62:42951
--builtin_ntp_servers=127.29.234.20:37549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 16:12:44.492548 3250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:44.492833 3250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:44.492924 3250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:44.500000 3250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:44.500092 3250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:44.500201 3250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.3
I20260924 16:12:44.509032 3250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:37549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266323169505-30632-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266323169505-30632-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.234.3
--webserver_port=0
--tserver_master_addrs=127.29.234.62:42951
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.3
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266323169505-30632-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:12:44.510339 3250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:44.512102 3250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:44.525331 3255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:44.525380 3256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:44.525558 3258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:44.526260 3250 server_base.cc:1034] running on GCE node
I20260924 16:12:44.526773 3250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:44.527688 3250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:44.528883 3250 hybrid_clock.cc:648] HybridClock initialized: now 1790266364528835 us; error 41 us; skew 500 ppm
I20260924 16:12:44.529206 3250 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 16:12:44.531441 3250 webserver.cc:460] Webserver started at http://127.29.234.3:33439/ using document root <none> and password file <none>
I20260924 16:12:44.532133 3250 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:44.532250 3250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:44.532603 3250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:44.535465 3250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266323169505-30632-0/minicluster-data/ts-2/data/instance:
uuid: "47e41fb850334a3a9fccf30aa0fa3355"
format_stamp: "Formatted at 2026-09-24 16:12:44 on dist-test-slave-v5j9"
server_key: "516fbffc76860a7560155c335b6bf35c"
server_key_iv: "d1a5058fc6d6deb40614b3ecaf96364c"
server_key_version: "encryptionkey@0"
I20260924 16:12:44.536244 3250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266323169505-30632-0/minicluster-data/ts-2/wal/instance:
uuid: "47e41fb850334a3a9fccf30aa0fa3355"
format_stamp: "Formatted at 2026-09-24 16:12:44 on dist-test-slave-v5j9"
server_key: "516fbffc76860a7560155c335b6bf35c"
server_key_iv: "d1a5058fc6d6deb40614b3ecaf96364c"
server_key_version: "encryptionkey@0"
I20260924 16:12:44.541196 3250 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 16:12:44.544557 3264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:44.545918 3250 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:12:44.546098 3250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266323169505-30632-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266323169505-30632-0/minicluster-data/ts-2/wal
uuid: "47e41fb850334a3a9fccf30aa0fa3355"
format_stamp: "Formatted at 2026-09-24 16:12:44 on dist-test-slave-v5j9"
server_key: "516fbffc76860a7560155c335b6bf35c"
server_key_iv: "d1a5058fc6d6deb40614b3ecaf96364c"
server_key_version: "encryptionkey@0"
I20260924 16:12:44.546335 3250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266323169505-30632-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266323169505-30632-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266323169505-30632-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:44.551914 3250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:44.552708 3250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:44.552987 3250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:44.554046 3250 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:12:44.555449 3250 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:12:44.555570 3250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:44.555702 3250 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:12:44.555791 3250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:44.589037 3250 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.3:34081
I20260924 16:12:44.589082 3378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.3:34081 every 8 connection(s)
I20260924 16:12:44.590443 3250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
I20260924 16:12:44.598372 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 3250
I20260924 16:12:44.598495 3113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:44.593048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.3:46085 (local address 127.29.234.62:42951)
0924 16:12:44.593322 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:44.593328 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:44.594781 (+ 1453us) server_negotiation.cc:407] Connection header received
0924 16:12:44.596368 (+ 1587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:44.596371 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:44.596423 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:44.596513 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:44.597801 (+ 1288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:44.597805 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:44.597808 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:12:44.597822 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:44.597851 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:12:44.597862 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:44.597959 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:44.598198 (+ 239us) server_negotiation.cc:299] Negotiation successful
0924 16:12:44.598247 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260924 16:12:44.598541 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266323169505-30632-0/minicluster-data/ts-2/wal/instance
I20260924 16:12:44.599089 30632 external_mini_cluster.cc:1428] Setting key 7b4595d65cac205f4a3f76197141d976
I20260924 16:12:44.599736 3382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:44.593707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:42951 (local address 127.29.234.3:46085)
0924 16:12:44.594611 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:44.594654 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:44.596132 (+ 1478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:44.596717 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:44.596725 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:44.596815 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:44.597532 (+ 717us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:12:44.597534 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:44.597572 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:12:44.597656 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:44.597967 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:44.597979 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:44.597989 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 16:12:44.598282 (+ 293us) client_negotiation.cc:240] Negotiation successful
0924 16:12:44.598560 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":260,"threads_started":1}
I20260924 16:12:44.601172 3379 heartbeater.cc:347] Connected to a master server at 127.29.234.62:42951
I20260924 16:12:44.601531 3379 heartbeater.cc:464] Registering TS with master...
I20260924 16:12:44.602207 3379 heartbeater.cc:511] Master 127.29.234.62:42951 requested a full tablet report, sending...
I20260924 16:12:44.603585 2922 ts_manager.cc:194] Registered new tserver with Master: 47e41fb850334a3a9fccf30aa0fa3355 (127.29.234.3:34081)
I20260924 16:12:44.614604 30632 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:12:44.621608 3113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:44.619019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46480 (local address 127.29.234.62:42951)
0924 16:12:44.619190 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:44.619194 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:44.619812 (+ 618us) server_negotiation.cc:407] Connection header received
0924 16:12:44.619958 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:44.619962 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:44.620057 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:44.620138 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:44.621101 (+ 963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:44.621105 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:44.621108 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:12:44.621121 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:44.621149 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:12:44.621160 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:44.621250 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:44.621291 (+ 41us) server_negotiation.cc:299] Negotiation successful
0924 16:12:44.621332 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 16:12:44.625562 2922 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46480:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:12:44.632484 2922 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:12:44.661670 3393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:44.644473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.2:41553 (local address 127.0.0.1:54846)
0924 16:12:44.656833 (+ 12360us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:44.656858 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:44.656973 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:44.658571 (+ 1598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:44.658584 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:44.658759 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:44.659719 (+ 960us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:12:44.659722 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:44.659791 (+ 69us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:12:44.659883 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:44.660824 (+ 941us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:44.660838 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:44.660849 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 16:12:44.661062 (+ 213us) client_negotiation.cc:240] Negotiation successful
0924 16:12:44.661154 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12125,"thread_start_us":10880,"threads_started":1}
I20260924 16:12:44.662645 3395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:44.644842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54846 (local address 127.29.234.2:41553)
0924 16:12:44.655313 (+ 10471us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:44.655324 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:44.657138 (+ 1814us) server_negotiation.cc:407] Connection header received
0924 16:12:44.657240 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:44.657252 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:44.657629 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:44.657810 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:44.660448 (+ 2638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:44.660460 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:44.660478 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:12:44.660506 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:44.660571 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:12:44.660593 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:44.660759 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:44.662154 (+ 1395us) server_negotiation.cc:299] Negotiation successful
0924 16:12:44.662259 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10275,"thread_start_us":10147,"threads_started":1}
I20260924 16:12:44.663055 3394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:44.644549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37324 (local address 127.29.234.3:34081)
0924 16:12:44.660029 (+ 15480us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:44.660042 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:44.660086 (+ 44us) server_negotiation.cc:407] Connection header received
0924 16:12:44.660163 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:44.660170 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:44.660412 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:44.661099 (+ 687us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:44.662227 (+ 1128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:44.662235 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:44.662245 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:12:44.662256 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:44.662293 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:12:44.662306 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:44.662501 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:44.662568 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 16:12:44.662662 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15259,"thread_start_us":15036,"threads_started":1}
I20260924 16:12:44.663595 3392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:44.644405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.3:34081 (local address 127.0.0.1:37324)
0924 16:12:44.655904 (+ 11499us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:44.655969 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:44.656155 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:44.660639 (+ 4484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:44.660644 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:44.660669 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:44.661427 (+ 758us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:12:44.661431 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:44.661450 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:12:44.661500 (+ 50us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:44.662417 (+ 917us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:44.662422 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:44.662426 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 16:12:44.662550 (+ 124us) client_negotiation.cc:240] Negotiation successful
0924 16:12:44.662601 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11148,"thread_start_us":12029,"threads_started":1}
I20260924 16:12:44.665807 3397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:44.657418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40528 (local address 127.29.234.1:34175)
0924 16:12:44.660367 (+ 2949us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:44.660376 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:44.660982 (+ 606us) server_negotiation.cc:407] Connection header received
0924 16:12:44.661566 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:44.661577 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:44.661791 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:44.661949 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:44.664978 (+ 3029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:44.664986 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:44.664998 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:12:44.665017 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:44.665075 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:12:44.665100 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:44.665239 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:44.665397 (+ 158us) server_negotiation.cc:299] Negotiation successful
0924 16:12:44.665503 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2777,"thread_start_us":2650,"threads_started":1}
I20260924 16:12:44.665843 3396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:44.656796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:34175 (local address 127.0.0.1:40528)
0924 16:12:44.660778 (+ 3982us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:44.660800 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:44.661392 (+ 592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:44.662921 (+ 1529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:44.662929 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:44.662958 (+ 29us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:44.664753 (+ 1795us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:12:44.664756 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:44.664772 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:12:44.664816 (+ 44us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:44.665217 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:44.665223 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:44.665227 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 16:12:44.665397 (+ 170us) client_negotiation.cc:240] Negotiation successful
0924 16:12:44.665447 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3848,"thread_start_us":3707,"threads_started":1}
I20260924 16:12:44.667027 3179 tablet_service.cc:1467] Processing CreateTablet for tablet d1ff98e4215e40e788330297a62d9a1e (DEFAULT_TABLE table=test-table [id=365865f0f27c4917b9b38cdfc27629f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:12:44.667402 3313 tablet_service.cc:1467] Processing CreateTablet for tablet d1ff98e4215e40e788330297a62d9a1e (DEFAULT_TABLE table=test-table [id=365865f0f27c4917b9b38cdfc27629f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:12:44.668290 3179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1ff98e4215e40e788330297a62d9a1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:44.668833 3313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1ff98e4215e40e788330297a62d9a1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:44.670288 3045 tablet_service.cc:1467] Processing CreateTablet for tablet d1ff98e4215e40e788330297a62d9a1e (DEFAULT_TABLE table=test-table [id=365865f0f27c4917b9b38cdfc27629f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:12:44.671581 3045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1ff98e4215e40e788330297a62d9a1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:44.679585 3398 tablet_bootstrap.cc:492] T d1ff98e4215e40e788330297a62d9a1e P c2696379a76c4f448e58c70cb3044f70: Bootstrap starting.
I20260924 16:12:44.682744 3398 tablet_bootstrap.cc:654] T d1ff98e4215e40e788330297a62d9a1e P c2696379a76c4f448e58c70cb3044f70: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:44.683893 3398 log.cc:824] T d1ff98e4215e40e788330297a62d9a1e P c2696379a76c4f448e58c70cb3044f70: Log is configured to *not* fsync() on all Append() calls
I20260924 16:12:44.684175 3399 tablet_bootstrap.cc:492] T d1ff98e4215e40e788330297a62d9a1e P 47e41fb850334a3a9fccf30aa0fa3355: Bootstrap starting.
I20260924 16:12:44.687989 3399 tablet_bootstrap.cc:654] T d1ff98e4215e40e788330297a62d9a1e P 47e41fb850334a3a9fccf30aa0fa3355: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:44.688031 3398 tablet_bootstrap.cc:492] T d1ff98e4215e40e788330297a62d9a1e P c2696379a76c4f448e58c70cb3044f70: No bootstrap required, opened a new log
I20260924 16:12:44.688289 3398 ts_tablet_manager.cc:1397] T d1ff98e4215e40e788330297a62d9a1e P c2696379a76c4f448e58c70cb3044f70: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.005s
I20260924 16:12:44.689200 3399 log.cc:824] T d1ff98e4215e40e788330297a62d9a1e P 47e41fb850334a3a9fccf30aa0fa3355: Log is configured to *not* fsync() on all Append() calls
I20260924 16:12:44.692271 3399 tablet_bootstrap.cc:492] T d1ff98e4215e40e788330297a62d9a1e P 47e41fb850334a3a9fccf30aa0fa3355: No bootstrap required, opened a new log
I20260924 16:12:44.692452 3401 tablet_bootstrap.cc:492] T d1ff98e4215e40e788330297a62d9a1e P 0dc415726f1d4d329d599ae69300b95a: Bootstrap starting.
I20260924 16:12:44.692534 3399 ts_tablet_manager.cc:1397] T d1ff98e4215e40e788330297a62d9a1e P 47e41fb850334a3a9fccf30aa0fa3355: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260924 16:12:44.695472 3401 tablet_bootstrap.cc:654] T d1ff98e4215e40e788330297a62d9a1e P 0dc415726f1d4d329d599ae69300b95a: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:44.696401 3401 log.cc:824] T d1ff98e4215e40e788330297a62d9a1e P 0dc415726f1d4d329d599ae69300b95a: Log is configured to *not* fsync() on all Append() calls
I20260924 16:12:44.698936 3401 tablet_bootstrap.cc:492] T d1ff98e4215e40e788330297a62d9a1e P 0dc415726f1d4d329d599ae69300b95a: No bootstrap required, opened a new log
I20260924 16:12:44.698825 3398 raft_consensus.cc:348] T d1ff98e4215e40e788330297a62d9a1e P c2696379a76c4f448e58c70cb3044f70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47e41fb850334a3a9fccf30aa0fa3355" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 34081 } } peers { permanent_uuid: "c2696379a76c4f448e58c70cb3044f70" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41553 } } peers { permanent_uuid: "0dc415726f1d4d329d599ae69300b95a" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 34175 } }
I20260924 16:12:44.699219 3401 ts_tablet_manager.cc:1397] T d1ff98e4215e40e788330297a62d9a1e P 0dc415726f1d4d329d599ae69300b95a: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260924 16:12:44.699265 3398 raft_consensus.cc:374] T d1ff98e4215e40e788330297a62d9a1e P c2696379a76c4f448e58c70cb3044f70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:44.699399 3398 raft_consensus.cc:729] T d1ff98e4215e40e788330297a62d9a1e P c2696379a76c4f448e58c70cb3044f70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2696379a76c4f448e58c70cb3044f70, State: Initialized, Role: FOLLOWER
I20260924 16:12:44.700043 3398 consensus_queue.cc:260] T d1ff98e4215e40e788330297a62d9a1e P c2696379a76c4f448e58c70cb3044f70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47e41fb850334a3a9fccf30aa0fa3355" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 34081 } } peers { permanent_uuid: "c2696379a76c4f448e58c70cb3044f70" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41553 } } peers { permanent_uuid: "0dc415726f1d4d329d599ae69300b95a" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 34175 } }
I20260924 16:12:44.701107 3247 heartbeater.cc:503] Master 127.29.234.62:42951 was elected leader, sending a full tablet report...
I20260924 16:12:44.701818 3398 ts_tablet_manager.cc:1428] T d1ff98e4215e40e788330297a62d9a1e P c2696379a76c4f448e58c70cb3044f70: Time spent starting tablet: real 0.013s user 0.015s sys 0.000s
I20260924 16:12:44.703620 3399 raft_consensus.cc:348] T d1ff98e4215e40e788330297a62d9a1e P 47e41fb850334a3a9fccf30aa0fa3355 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47e41fb850334a3a9fccf30aa0fa3355" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 34081 } } peers { permanent_uuid: "c2696379a76c4f448e58c70cb3044f70" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41553 } } peers { permanent_uuid: "0dc415726f1d4d329d599ae69300b95a" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 34175 } }
I20260924 16:12:44.704063 3399 raft_consensus.cc:374] T d1ff98e4215e40e788330297a62d9a1e P 47e41fb850334a3a9fccf30aa0fa3355 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:44.704176 3399 raft_consensus.cc:729] T d1ff98e4215e40e788330297a62d9a1e P 47e41fb850334a3a9fccf30aa0fa3355 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47e41fb850334a3a9fccf30aa0fa3355, State: Initialized, Role: FOLLOWER
I20260924 16:12:44.704834 3399 consensus_queue.cc:260] T d1ff98e4215e40e788330297a62d9a1e P 47e41fb850334a3a9fccf30aa0fa3355 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47e41fb850334a3a9fccf30aa0fa3355" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 34081 } } peers { permanent_uuid: "c2696379a76c4f448e58c70cb3044f70" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41553 } } peers { permanent_uuid: "0dc415726f1d4d329d599ae69300b95a" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 34175 } }
I20260924 16:12:44.705986 3379 heartbeater.cc:503] Master 127.29.234.62:42951 was elected leader, sending a full tablet report...
I20260924 16:12:44.706765 3399 ts_tablet_manager.cc:1428] T d1ff98e4215e40e788330297a62d9a1e P 47e41fb850334a3a9fccf30aa0fa3355: Time spent starting tablet: real 0.014s user 0.010s sys 0.002s
I20260924 16:12:44.709198 3401 raft_consensus.cc:348] T d1ff98e4215e40e788330297a62d9a1e P 0dc415726f1d4d329d599ae69300b95a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47e41fb850334a3a9fccf30aa0fa3355" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 34081 } } peers { permanent_uuid: "c2696379a76c4f448e58c70cb3044f70" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41553 } } peers { permanent_uuid: "0dc415726f1d4d329d599ae69300b95a" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 34175 } }
I20260924 16:12:44.709678 3401 raft_consensus.cc:374] T d1ff98e4215e40e788330297a62d9a1e P 0dc415726f1d4d329d599ae69300b95a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:44.709786 3401 raft_consensus.cc:729] T d1ff98e4215e40e788330297a62d9a1e P 0dc415726f1d4d329d599ae69300b95a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dc415726f1d4d329d599ae69300b95a, State: Initialized, Role: FOLLOWER
I20260924 16:12:44.710458 3401 consensus_queue.cc:260] T d1ff98e4215e40e788330297a62d9a1e P 0dc415726f1d4d329d599ae69300b95a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47e41fb850334a3a9fccf30aa0fa3355" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 34081 } } peers { permanent_uuid: "c2696379a76c4f448e58c70cb3044f70" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41553 } } peers { permanent_uuid: "0dc415726f1d4d329d599ae69300b95a" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 34175 } }
I20260924 16:12:44.711493 3111 heartbeater.cc:503] Master 127.29.234.62:42951 was elected leader, sending a full tablet report...
I20260924 16:12:44.712154 3401 ts_tablet_manager.cc:1428] T d1ff98e4215e40e788330297a62d9a1e P 0dc415726f1d4d329d599ae69300b95a: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
W20260924 16:12:44.716091 3112 tablet.cc:2369] T d1ff98e4215e40e788330297a62d9a1e P 0dc415726f1d4d329d599ae69300b95a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:12:44.799018 3406 raft_consensus.cc:482] T d1ff98e4215e40e788330297a62d9a1e P 0dc415726f1d4d329d599ae69300b95a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:12:44.799389 3406 raft_consensus.cc:504] T d1ff98e4215e40e788330297a62d9a1e P 0dc415726f1d4d329d599ae69300b95a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47e41fb850334a3a9fccf30aa0fa3355" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 34081 } } peers { permanent_uuid: "c2696379a76c4f448e58c70cb3044f70" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41553 } } peers { permanent_uuid: "0dc415726f1d4d329d599ae69300b95a" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 34175 } }
I20260924 16:12:44.801226 3406 leader_election.cc:288] T d1ff98e4215e40e788330297a62d9a1e P 0dc415726f1d4d329d599ae69300b95a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47e41fb850334a3a9fccf30aa0fa3355 (127.29.234.3:34081), c2696379a76c4f448e58c70cb3044f70 (127.29.234.2:41553)
I20260924 16:12:44.804836 3394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:44.801615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.1:55139 (local address 127.29.234.3:34081)
0924 16:12:44.801839 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:44.801847 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:44.802811 (+ 964us) server_negotiation.cc:407] Connection header received
0924 16:12:44.803007 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:44.803014 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:44.803071 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:44.803174 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:44.804212 (+ 1038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:44.804218 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:44.804223 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:12:44.804241 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:44.804281 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:12:44.804293 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:44.804427 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:44.804504 (+ 77us) server_negotiation.cc:299] Negotiation successful
0924 16:12:44.804568 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 16:12:44.805004 3395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:44.801815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.1:43879 (local address 127.29.234.2:41553)
0924 16:12:44.802069 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:44.802079 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:44.803018 (+ 939us) server_negotiation.cc:407] Connection header received
0924 16:12:44.803193 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:44.803197 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:44.803244 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:44.803352 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:44.804390 (+ 1038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:44.804394 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:44.804397 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:12:44.804412 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:44.804443 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:12:44.804453 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:44.804580 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:44.804658 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 16:12:44.804725 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260924 16:12:44.805058 3407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:44.801658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.3:34081 (local address 127.29.234.1:55139)
0924 16:12:44.802692 (+ 1034us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:44.802707 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:44.802862 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:44.803203 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:44.803209 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:44.803233 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:44.803966 (+ 733us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:12:44.803971 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:44.803991 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:12:44.804043 (+ 52us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:44.804393 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:44.804397 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:44.804404 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 16:12:44.804544 (+ 140us) client_negotiation.cc:240] Negotiation successful
0924 16:12:44.804595 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":904,"thread_start_us":663,"threads_started":1}
I20260924 16:12:44.805250 3408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:44.802091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.2:41553 (local address 127.29.234.1:43879)
0924 16:12:44.802899 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:44.802918 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:44.803050 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:44.803384 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:44.803390 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:44.803413 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:44.804174 (+ 761us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:12:44.804176 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:44.804189 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:12:44.804224 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:44.804549 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:44.804553 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:44.804560 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 16:12:44.804702 (+ 142us) client_negotiation.cc:240] Negotiation successful
0924 16:12:44.804753 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"thread_start_us":482,"threads_started":1}
I20260924 16:12:44.806136 3333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1ff98e4215e40e788330297a62d9a1e" candidate_uuid: "0dc415726f1d4d329d599ae69300b95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47e41fb850334a3a9fccf30aa0fa3355" is_pre_election: true
I20260924 16:12:44.806270 3200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1ff98e4215e40e788330297a62d9a1e" candidate_uuid: "0dc415726f1d4d329d599ae69300b95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2696379a76c4f448e58c70cb3044f70" is_pre_election: true
I20260924 16:12:44.806694 3333 raft_consensus.cc:2454] T d1ff98e4215e40e788330297a62d9a1e P 47e41fb850334a3a9fccf30aa0fa3355 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0dc415726f1d4d329d599ae69300b95a in term 0.
I20260924 16:12:44.806834 3200 raft_consensus.cc:2454] T d1ff98e4215e40e788330297a62d9a1e P c2696379a76c4f448e58c70cb3044f70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0dc415726f1d4d329d599ae69300b95a in term 0.
I20260924 16:12:44.807436 2996 leader_election.cc:302] T d1ff98e4215e40e788330297a62d9a1e P 0dc415726f1d4d329d599ae69300b95a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dc415726f1d4d329d599ae69300b95a, 47e41fb850334a3a9fccf30aa0fa3355; no voters:
I20260924 16:12:44.807862 3406 raft_consensus.cc:2781] T d1ff98e4215e40e788330297a62d9a1e P 0dc415726f1d4d329d599ae69300b95a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:12:44.808009 3406 raft_consensus.cc:482] T d1ff98e4215e40e788330297a62d9a1e P 0dc415726f1d4d329d599ae69300b95a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:12:44.808127 3406 raft_consensus.cc:3037] T d1ff98e4215e40e788330297a62d9a1e P 0dc415726f1d4d329d599ae69300b95a [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:44.810461 3406 raft_consensus.cc:504] T d1ff98e4215e40e788330297a62d9a1e P 0dc415726f1d4d329d599ae69300b95a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47e41fb850334a3a9fccf30aa0fa3355" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 34081 } } peers { permanent_uuid: "c2696379a76c4f448e58c70cb3044f70" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41553 } } peers { permanent_uuid: "0dc415726f1d4d329d599ae69300b95a" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 34175 } }
I20260924 16:12:44.811084 3406 leader_election.cc:288] T d1ff98e4215e40e788330297a62d9a1e P 0dc415726f1d4d329d599ae69300b95a [CANDIDATE]: Term 1 election: Requested vote from peers 47e41fb850334a3a9fccf30aa0fa3355 (127.29.234.3:34081), c2696379a76c4f448e58c70cb3044f70 (127.29.234.2:41553)
I20260924 16:12:44.811560 3333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1ff98e4215e40e788330297a62d9a1e" candidate_uuid: "0dc415726f1d4d329d599ae69300b95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47e41fb850334a3a9fccf30aa0fa3355"
I20260924 16:12:44.811560 3200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1ff98e4215e40e788330297a62d9a1e" candidate_uuid: "0dc415726f1d4d329d599ae69300b95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2696379a76c4f448e58c70cb3044f70"
I20260924 16:12:44.811789 3200 raft_consensus.cc:3037] T d1ff98e4215e40e788330297a62d9a1e P c2696379a76c4f448e58c70cb3044f70 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:44.811777 3333 raft_consensus.cc:3037] T d1ff98e4215e40e788330297a62d9a1e P 47e41fb850334a3a9fccf30aa0fa3355 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:44.815096 3333 raft_consensus.cc:2454] T d1ff98e4215e40e788330297a62d9a1e P 47e41fb850334a3a9fccf30aa0fa3355 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0dc415726f1d4d329d599ae69300b95a in term 1.
I20260924 16:12:44.815096 3200 raft_consensus.cc:2454] T d1ff98e4215e40e788330297a62d9a1e P c2696379a76c4f448e58c70cb3044f70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0dc415726f1d4d329d599ae69300b95a in term 1.
I20260924 16:12:44.815577 2998 leader_election.cc:302] T d1ff98e4215e40e788330297a62d9a1e P 0dc415726f1d4d329d599ae69300b95a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dc415726f1d4d329d599ae69300b95a, c2696379a76c4f448e58c70cb3044f70; no voters:
I20260924 16:12:44.815922 3406 raft_consensus.cc:2781] T d1ff98e4215e40e788330297a62d9a1e P 0dc415726f1d4d329d599ae69300b95a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:12:44.816483 3406 raft_consensus.cc:686] T d1ff98e4215e40e788330297a62d9a1e P 0dc415726f1d4d329d599ae69300b95a [term 1 LEADER]: Becoming Leader. State: Replica: 0dc415726f1d4d329d599ae69300b95a, State: Running, Role: LEADER
I20260924 16:12:44.817092 3406 consensus_queue.cc:237] T d1ff98e4215e40e788330297a62d9a1e P 0dc415726f1d4d329d599ae69300b95a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47e41fb850334a3a9fccf30aa0fa3355" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 34081 } } peers { permanent_uuid: "c2696379a76c4f448e58c70cb3044f70" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41553 } } peers { permanent_uuid: "0dc415726f1d4d329d599ae69300b95a" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 34175 } }
I20260924 16:12:44.822419 2922 catalog_manager.cc:5495] T d1ff98e4215e40e788330297a62d9a1e P 0dc415726f1d4d329d599ae69300b95a reported cstate change: term changed from 0 to 1, leader changed from <none> to 0dc415726f1d4d329d599ae69300b95a (127.29.234.1). New cstate: current_term: 1 leader_uuid: "0dc415726f1d4d329d599ae69300b95a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47e41fb850334a3a9fccf30aa0fa3355" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 34081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2696379a76c4f448e58c70cb3044f70" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0dc415726f1d4d329d599ae69300b95a" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 34175 } health_report { overall_health: HEALTHY } } }
W20260924 16:12:44.842239 3381 tablet.cc:2369] T d1ff98e4215e40e788330297a62d9a1e P 47e41fb850334a3a9fccf30aa0fa3355: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:12:44.863135 3397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:44.861032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40538 (local address 127.29.234.1:34175)
0924 16:12:44.861285 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:44.861290 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:44.861310 (+ 20us) server_negotiation.cc:407] Connection header received
0924 16:12:44.861553 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:44.861557 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:44.861640 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:44.861723 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:44.862531 (+ 808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:44.862535 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:44.862538 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:12:44.862553 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:44.862585 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:12:44.862596 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:44.862685 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:44.862781 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 16:12:44.862828 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260924 16:12:44.871328 3200 raft_consensus.cc:1261] T d1ff98e4215e40e788330297a62d9a1e P c2696379a76c4f448e58c70cb3044f70 [term 1 FOLLOWER]: Refusing update from remote peer 0dc415726f1d4d329d599ae69300b95a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:12:44.871333 3333 raft_consensus.cc:1261] T d1ff98e4215e40e788330297a62d9a1e P 47e41fb850334a3a9fccf30aa0fa3355 [term 1 FOLLOWER]: Refusing update from remote peer 0dc415726f1d4d329d599ae69300b95a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:12:44.872257 3406 consensus_queue.cc:1035] T d1ff98e4215e40e788330297a62d9a1e P 0dc415726f1d4d329d599ae69300b95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2696379a76c4f448e58c70cb3044f70" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:12:44.872576 3409 consensus_queue.cc:1035] T d1ff98e4215e40e788330297a62d9a1e P 0dc415726f1d4d329d599ae69300b95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "47e41fb850334a3a9fccf30aa0fa3355" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 34081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:12:44.901268 2922 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:12:44.901635 2922 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:12:44.905066 2922 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3e1e4efebe5c4938b6a0517f7afd8820: Sending DeleteTablet for 3 replicas of tablet d1ff98e4215e40e788330297a62d9a1e
I20260924 16:12:44.906280 3313 tablet_service.cc:1514] Processing DeleteTablet for tablet d1ff98e4215e40e788330297a62d9a1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:12:44 UTC) from {username='slave'} at 127.0.0.1:37324
I20260924 16:12:44.906733 3045 tablet_service.cc:1514] Processing DeleteTablet for tablet d1ff98e4215e40e788330297a62d9a1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:12:44 UTC) from {username='slave'} at 127.0.0.1:40528
I20260924 16:12:44.906749 3179 tablet_service.cc:1514] Processing DeleteTablet for tablet d1ff98e4215e40e788330297a62d9a1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:12:44 UTC) from {username='slave'} at 127.0.0.1:54846
I20260924 16:12:44.908437 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 2981
I20260924 16:12:44.936851 3424 tablet_replica.cc:330] T d1ff98e4215e40e788330297a62d9a1e P c2696379a76c4f448e58c70cb3044f70: stopping tablet replica
I20260924 16:12:44.938697 3397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:44.909728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40552 (local address 127.29.234.1:34175)
0924 16:12:44.909959 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:44.909968 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:44.934672 (+ 24704us) server_negotiation.cc:407] Connection header received
0924 16:12:44.935311 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:44.935321 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:44.935414 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:44.935510 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:44.937549 (+ 2039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:44.937558 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:44.937566 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:12:44.937592 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:44.937645 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:12:44.937670 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:44.937809 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:44.938125 (+ 316us) server_negotiation.cc:299] Negotiation successful
0924 16:12:44.938206 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 16:12:44.940439 3106 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:12:44.940888 3424 raft_consensus.cc:2229] T d1ff98e4215e40e788330297a62d9a1e P c2696379a76c4f448e58c70cb3044f70 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:12:44.941668 3424 raft_consensus.cc:2258] T d1ff98e4215e40e788330297a62d9a1e P c2696379a76c4f448e58c70cb3044f70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:12:44.942574 3422 tablet_replica.cc:330] T d1ff98e4215e40e788330297a62d9a1e P 47e41fb850334a3a9fccf30aa0fa3355: stopping tablet replica
I20260924 16:12:44.943483 3422 raft_consensus.cc:2229] T d1ff98e4215e40e788330297a62d9a1e P 47e41fb850334a3a9fccf30aa0fa3355 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:12:44.946732 3424 ts_tablet_manager.cc:1905] T d1ff98e4215e40e788330297a62d9a1e P c2696379a76c4f448e58c70cb3044f70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:12:44.947047 3422 raft_consensus.cc:2258] T d1ff98e4215e40e788330297a62d9a1e P 47e41fb850334a3a9fccf30aa0fa3355 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:12:44.949111 3422 ts_tablet_manager.cc:1905] T d1ff98e4215e40e788330297a62d9a1e P 47e41fb850334a3a9fccf30aa0fa3355: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:12:44.954082 3422 ts_tablet_manager.cc:1918] T d1ff98e4215e40e788330297a62d9a1e P 47e41fb850334a3a9fccf30aa0fa3355: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:12:44.954263 3422 log.cc:1196] T d1ff98e4215e40e788330297a62d9a1e P 47e41fb850334a3a9fccf30aa0fa3355: Deleting WAL directory at /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266323169505-30632-0/minicluster-data/ts-2/wal/wals/d1ff98e4215e40e788330297a62d9a1e
I20260924 16:12:44.954753 3422 ts_tablet_manager.cc:1939] T d1ff98e4215e40e788330297a62d9a1e P 47e41fb850334a3a9fccf30aa0fa3355: Deleting consensus metadata
I20260924 16:12:44.955411 3424 ts_tablet_manager.cc:1918] T d1ff98e4215e40e788330297a62d9a1e P c2696379a76c4f448e58c70cb3044f70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:12:44.955575 3424 log.cc:1196] T d1ff98e4215e40e788330297a62d9a1e P c2696379a76c4f448e58c70cb3044f70: Deleting WAL directory at /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790266323169505-30632-0/minicluster-data/ts-1/wal/wals/d1ff98e4215e40e788330297a62d9a1e
I20260924 16:12:44.956178 3424 ts_tablet_manager.cc:1939] T d1ff98e4215e40e788330297a62d9a1e P c2696379a76c4f448e58c70cb3044f70: Deleting consensus metadata
I20260924 16:12:44.956467 2905 catalog_manager.cc:4841] TS 47e41fb850334a3a9fccf30aa0fa3355 (127.29.234.3:34081): tablet d1ff98e4215e40e788330297a62d9a1e (table test-table [id=365865f0f27c4917b9b38cdfc27629f6]) successfully deleted
I20260924 16:12:44.958118 2907 catalog_manager.cc:4841] TS c2696379a76c4f448e58c70cb3044f70 (127.29.234.2:41553): tablet d1ff98e4215e40e788330297a62d9a1e (table test-table [id=365865f0f27c4917b9b38cdfc27629f6]) successfully deleted
I20260924 16:12:45.043520 3423 tablet_replica.cc:330] T d1ff98e4215e40e788330297a62d9a1e P 0dc415726f1d4d329d599ae69300b95a: stopping tablet replica
I20260924 16:12:45.043804 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 2981
W20260924 16:12:45.062790 2907 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.234.1:34175 (error 108)
I20260924 16:12:45.063081 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 3115
I20260924 16:12:45.064985 3396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:45.064500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:34175 (local address 127.0.0.1:40564)
0924 16:12:45.064733 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:45.064851 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.1:34175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260924 16:12:45.065304 2907 catalog_manager.cc:4584] TS 0dc415726f1d4d329d599ae69300b95a (127.29.234.1:34175): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d1ff98e4215e40e788330297a62d9a1e: Network error: Client connection negotiation failed: client connection to 127.29.234.1:34175: connect: Connection refused (error 111)
I20260924 16:12:45.066073 3395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:45.064042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54858 (local address 127.29.234.2:41553)
0924 16:12:45.064234 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:45.064238 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:45.064262 (+ 24us) server_negotiation.cc:407] Connection header received
0924 16:12:45.064364 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:45.064368 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:45.064452 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:45.064539 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:45.065549 (+ 1010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:45.065555 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:45.065558 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:12:45.065576 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:45.065615 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:12:45.065632 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:45.065759 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:45.065833 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 16:12:45.065877 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 16:12:45.066695 3242 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:12:45.119112 3396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:45.118516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:34175 (local address 127.0.0.1:40578)
0924 16:12:45.118783 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:45.118895 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.1:34175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260924 16:12:45.120249 3396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:45.119806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:34175 (local address 127.0.0.1:40594)
0924 16:12:45.119983 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:45.120080 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.1:34175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260924 16:12:45.154527 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 3115
I20260924 16:12:45.173436 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 3250
I20260924 16:12:45.176174 3394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:45.174287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37334 (local address 127.29.234.3:34081)
0924 16:12:45.174443 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:45.174448 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:45.174467 (+ 19us) server_negotiation.cc:407] Connection header received
0924 16:12:45.174635 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:45.174638 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:45.174721 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:45.174802 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:45.175589 (+ 787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:45.175593 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:45.175596 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:12:45.175609 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:45.175640 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:12:45.175651 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:45.175753 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:45.175811 (+ 58us) server_negotiation.cc:299] Negotiation successful
0924 16:12:45.175865 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 16:12:45.176779 3374 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:12:45.181064 3396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:45.180667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:34175 (local address 127.0.0.1:40600)
0924 16:12:45.180847 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:45.180937 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.1:34175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260924 16:12:45.182195 3396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:45.181821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:34175 (local address 127.0.0.1:40602)
0924 16:12:45.181964 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:45.182077 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.1:34175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 16:12:45.262993 3396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:45.262448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:34175 (local address 127.0.0.1:40608)
0924 16:12:45.262704 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:45.262812 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.1:34175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260924 16:12:45.264097 3396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:45.263689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:34175 (local address 127.0.0.1:40616)
0924 16:12:45.263911 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:45.263977 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.1:34175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260924 16:12:45.274428 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 3250
I20260924 16:12:45.292014 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 2889
I20260924 16:12:45.293442 2952 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:12:45.399012 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 2889
2026-09-24T16:12:45Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2684 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260924 16:12:45.442157 30632 mini_oidc.cc:152] Starting JWKS server
I20260924 16:12:45.444916 30632 webserver.cc:460] Webserver started at https://127.0.0.1:37119/ using document root <none> and password file <none>
I20260924 16:12:45.445230 30632 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 16:12:45.445907 30632 webserver.cc:460] Webserver started at http://127.0.0.1:34391/ using document root <none> and password file <none>
2026-09-24T16:12:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:12:45Z Disabled control of system clock
I20260924 16:12:45.472946 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790266323169505-30632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790266323169505-30632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790266323169505-30632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:41621
--webserver_interface=127.29.234.62
--webserver_port=0
--builtin_ntp_servers=127.29.234.20:39737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.234.62:41621
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37119/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790266323169505-30632-0/testchainca.cert with env {}
W20260924 16:12:45.798379 3436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:45.798732 3436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:45.798882 3436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:45.806048 3436 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:12:45.806212 3436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:45.806290 3436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:45.806375 3436 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:12:45.806454 3436 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 16:12:45.806565 3436 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 16:12:45.806640 3436 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37119/jwks.json
I20260924 16:12:45.816557 3436 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:39737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790266323169505-30632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790266323169505-30632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.234.62:41621
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:41621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37119/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.234.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790266323169505-30632-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790266323169505-30632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:12:45.817937 3436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:45.819397 3436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:45.831284 3444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:45.831346 3442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:45.831818 3441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:45.831928 3436 server_base.cc:1034] running on GCE node
I20260924 16:12:45.832765 3436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:45.834247 3436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:45.835485 3436 hybrid_clock.cc:648] HybridClock initialized: now 1790266365835442 us; error 36 us; skew 500 ppm
I20260924 16:12:45.835850 3436 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 16:12:45.838809 3436 webserver.cc:460] Webserver started at http://127.29.234.62:46483/ using document root <none> and password file <none>
I20260924 16:12:45.839864 3436 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:45.840000 3436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:45.840420 3436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:45.844278 3436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790266323169505-30632-0/minicluster-data/master-0/data/instance:
uuid: "0b46a548ab3b41e08f75253a4536f131"
format_stamp: "Formatted at 2026-09-24 16:12:45 on dist-test-slave-v5j9"
server_key: "897aacc997277a56f8dd6c06694fc9c1"
server_key_iv: "2fa0e271cff8f07c188e2bad454c2ef0"
server_key_version: "encryptionkey@0"
I20260924 16:12:45.845379 3436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790266323169505-30632-0/minicluster-data/master-0/wal/instance:
uuid: "0b46a548ab3b41e08f75253a4536f131"
format_stamp: "Formatted at 2026-09-24 16:12:45 on dist-test-slave-v5j9"
server_key: "897aacc997277a56f8dd6c06694fc9c1"
server_key_iv: "2fa0e271cff8f07c188e2bad454c2ef0"
server_key_version: "encryptionkey@0"
I20260924 16:12:45.852041 3436 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.003s
I20260924 16:12:45.855461 3450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:45.856906 3436 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 16:12:45.857121 3436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790266323169505-30632-0/minicluster-data/master-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790266323169505-30632-0/minicluster-data/master-0/wal
uuid: "0b46a548ab3b41e08f75253a4536f131"
format_stamp: "Formatted at 2026-09-24 16:12:45 on dist-test-slave-v5j9"
server_key: "897aacc997277a56f8dd6c06694fc9c1"
server_key_iv: "2fa0e271cff8f07c188e2bad454c2ef0"
server_key_version: "encryptionkey@0"
I20260924 16:12:45.857358 3436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790266323169505-30632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790266323169505-30632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790266323169505-30632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:45.878321 3436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:45.879187 3436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:45.879508 3436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:45.897822 3436 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.62:41621
I20260924 16:12:45.897845 3503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.62:41621 every 8 connection(s)
I20260924 16:12:45.899209 3436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
I20260924 16:12:45.900520 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 3436
I20260924 16:12:45.900735 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790266323169505-30632-0/minicluster-data/master-0/wal/instance
I20260924 16:12:45.901568 30632 external_mini_cluster.cc:1428] Setting key a35086e3bd0d507cd2f7462c4365e3eb
I20260924 16:12:45.904057 3504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:45.915488 3507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:45.903543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39216 (local address 127.29.234.62:41621)
0924 16:12:45.904536 (+ 993us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:45.904547 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:45.904590 (+ 43us) server_negotiation.cc:407] Connection header received
0924 16:12:45.906285 (+ 1695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:45.906316 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:45.906795 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:45.907141 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:45.907839 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:45.909197 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:45.911086 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:45.911722 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:45.911860 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:45.913133 (+ 1273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:45.913141 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:45.913154 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:12:45.913772 (+ 618us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:45.913857 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:12:45.913877 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:45.913970 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:45.914205 (+ 235us) server_negotiation.cc:299] Negotiation successful
0924 16:12:45.914555 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":358,"threads_started":1}
I20260924 16:12:45.916963 3504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b46a548ab3b41e08f75253a4536f131: Bootstrap starting.
I20260924 16:12:45.919451 30632 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:12:45.920399 3504 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b46a548ab3b41e08f75253a4536f131: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:45.921444 3504 log.cc:824] T 00000000000000000000000000000000 P 0b46a548ab3b41e08f75253a4536f131: Log is configured to *not* fsync() on all Append() calls
I20260924 16:12:45.924402 3504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b46a548ab3b41e08f75253a4536f131: No bootstrap required, opened a new log
I20260924 16:12:45.931653 3504 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b46a548ab3b41e08f75253a4536f131 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b46a548ab3b41e08f75253a4536f131" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 41621 } }
I20260924 16:12:45.932169 3504 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b46a548ab3b41e08f75253a4536f131 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:45.932300 3504 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b46a548ab3b41e08f75253a4536f131 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b46a548ab3b41e08f75253a4536f131, State: Initialized, Role: FOLLOWER
I20260924 16:12:45.932968 3504 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b46a548ab3b41e08f75253a4536f131 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b46a548ab3b41e08f75253a4536f131" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 41621 } }
I20260924 16:12:45.933207 3504 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b46a548ab3b41e08f75253a4536f131 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:12:45.933331 3504 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b46a548ab3b41e08f75253a4536f131 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:12:45.933578 3504 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b46a548ab3b41e08f75253a4536f131 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:45.935685 3504 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b46a548ab3b41e08f75253a4536f131 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b46a548ab3b41e08f75253a4536f131" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 41621 } }
I20260924 16:12:45.936201 3504 leader_election.cc:302] T 00000000000000000000000000000000 P 0b46a548ab3b41e08f75253a4536f131 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b46a548ab3b41e08f75253a4536f131; no voters:
I20260924 16:12:45.936816 3504 leader_election.cc:288] T 00000000000000000000000000000000 P 0b46a548ab3b41e08f75253a4536f131 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:12:45.937024 3509 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b46a548ab3b41e08f75253a4536f131 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:12:45.937960 3509 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b46a548ab3b41e08f75253a4536f131 [term 1 LEADER]: Becoming Leader. State: Replica: 0b46a548ab3b41e08f75253a4536f131, State: Running, Role: LEADER
I20260924 16:12:45.938521 3509 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b46a548ab3b41e08f75253a4536f131 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b46a548ab3b41e08f75253a4536f131" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 41621 } }
I20260924 16:12:45.939020 3504 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b46a548ab3b41e08f75253a4536f131 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:12:45.941747 3511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b46a548ab3b41e08f75253a4536f131 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b46a548ab3b41e08f75253a4536f131. Latest consensus state: current_term: 1 leader_uuid: "0b46a548ab3b41e08f75253a4536f131" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b46a548ab3b41e08f75253a4536f131" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 41621 } } }
I20260924 16:12:45.941836 3510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b46a548ab3b41e08f75253a4536f131 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b46a548ab3b41e08f75253a4536f131" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b46a548ab3b41e08f75253a4536f131" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 41621 } } }
I20260924 16:12:45.942155 3511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b46a548ab3b41e08f75253a4536f131 [sys.catalog]: This master's current role is: LEADER
I20260924 16:12:45.942239 3510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b46a548ab3b41e08f75253a4536f131 [sys.catalog]: This master's current role is: LEADER
I20260924 16:12:45.946673 3516 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:12:45.952107 3516 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:12:45.961774 3516 catalog_manager.cc:1347] Generated new cluster ID: b8f37714efca45e6b4a40e7e669eb7d9
I20260924 16:12:45.961946 3516 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:12:45.985288 3516 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:12:45.986694 3516 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:12:45.993384 3516 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b46a548ab3b41e08f75253a4536f131: Generated new TSK 0
I20260924 16:12:45.994413 3516 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 16:12:46.040293 3535 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260924 16:12:46.040473 3535 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 16:12:46.042800 3507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:46.018410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39228 (local address 127.29.234.62:41621)
0924 16:12:46.018635 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:46.018642 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:46.019228 (+ 586us) server_negotiation.cc:407] Connection header received
0924 16:12:46.019541 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:46.019547 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:46.019644 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:46.019814 (+ 170us) server_negotiation.cc:226] Negotiated authn=JWT
0924 16:12:46.020378 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:46.021529 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:46.022991 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:46.023700 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:46.023829 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:46.024342 (+ 513us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 16:12:46.042078 (+ 17736us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37119/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 16:12:46.042452 (+ 374us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39228: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37119/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":77}
W20260924 16:12:46.042950 3507 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39228: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37119/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260924 16:12:46.044237 3533 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:12:46.018422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:41621 (local address 127.0.0.1:39228)
0924 16:12:46.019114 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:46.019130 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:46.019286 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:46.019839 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:46.019843 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:46.019853 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0924 16:12:46.020216 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:46.020225 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:46.021688 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:46.021692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:46.022851 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:46.022863 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:46.023880 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:46.023884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:46.024005 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:46.024064 (+ 59us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 16:12:46.042404 (+ 18340us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37119/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 16:12:46.042693 (+ 289us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.29.234.62:41621: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37119/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":353,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.29.234.62:41621: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37119/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260924 16:12:46.045972 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 3436
I20260924 16:12:46.047202 3498 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:12:46.144734 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 3436
2026-09-24T16:12:46Z chronyd exiting
I20260924 16:12:46.195648 30632 test_util.cc:175] -----------------------------------------------
I20260924 16:12:46.195811 30632 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790266323169505-30632-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (758 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260924 16:12:46.199483 30632 mini_oidc.cc:152] Starting JWKS server
I20260924 16:12:46.201195 30632 webserver.cc:460] Webserver started at https://127.0.0.1:39839/ using document root <none> and password file <none>
I20260924 16:12:46.201498 30632 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 16:12:46.202157 30632 webserver.cc:460] Webserver started at http://127.0.0.1:43773/ using document root <none> and password file <none>
2026-09-24T16:12:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:12:46Z Disabled control of system clock
I20260924 16:12:46.229812 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790266323169505-30632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790266323169505-30632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790266323169505-30632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:34775
--webserver_interface=127.29.234.62
--webserver_port=0
--builtin_ntp_servers=127.29.234.20:40659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.234.62:34775
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39839/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790266323169505-30632-0/testchainca.cert with env {}
W20260924 16:12:46.549247 3542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:46.549667 3542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:46.549821 3542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:46.557046 3542 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:12:46.557188 3542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:46.557278 3542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:46.557407 3542 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:12:46.557495 3542 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 16:12:46.557585 3542 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 16:12:46.557660 3542 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39839/jwks.json
I20260924 16:12:46.567404 3542 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:40659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790266323169505-30632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790266323169505-30632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.234.62:34775
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:34775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39839/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.234.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790266323169505-30632-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790266323169505-30632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:12:46.568689 3542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:46.570065 3542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:46.581486 3547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:46.581533 3550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:46.581557 3548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:46.582697 3542 server_base.cc:1034] running on GCE node
I20260924 16:12:46.583441 3542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:46.584734 3542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:46.585963 3542 hybrid_clock.cc:648] HybridClock initialized: now 1790266366585933 us; error 57 us; skew 500 ppm
I20260924 16:12:46.586324 3542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 16:12:46.592670 3542 webserver.cc:460] Webserver started at http://127.29.234.62:44609/ using document root <none> and password file <none>
I20260924 16:12:46.593451 3542 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:46.593587 3542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:46.593941 3542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:46.596845 3542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790266323169505-30632-0/minicluster-data/master-0/data/instance:
uuid: "fc0369203851432daadd1d5800ff7398"
format_stamp: "Formatted at 2026-09-24 16:12:46 on dist-test-slave-v5j9"
server_key: "1d55afc0b0d1fd4b74b2e7e97cc218db"
server_key_iv: "4e0d2964caa35c6d9ddedc0ae3361155"
server_key_version: "encryptionkey@0"
I20260924 16:12:46.597678 3542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790266323169505-30632-0/minicluster-data/master-0/wal/instance:
uuid: "fc0369203851432daadd1d5800ff7398"
format_stamp: "Formatted at 2026-09-24 16:12:46 on dist-test-slave-v5j9"
server_key: "1d55afc0b0d1fd4b74b2e7e97cc218db"
server_key_iv: "4e0d2964caa35c6d9ddedc0ae3361155"
server_key_version: "encryptionkey@0"
I20260924 16:12:46.602571 3542 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 16:12:46.606087 3556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:46.607493 3542 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:12:46.607682 3542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790266323169505-30632-0/minicluster-data/master-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790266323169505-30632-0/minicluster-data/master-0/wal
uuid: "fc0369203851432daadd1d5800ff7398"
format_stamp: "Formatted at 2026-09-24 16:12:46 on dist-test-slave-v5j9"
server_key: "1d55afc0b0d1fd4b74b2e7e97cc218db"
server_key_iv: "4e0d2964caa35c6d9ddedc0ae3361155"
server_key_version: "encryptionkey@0"
I20260924 16:12:46.607903 3542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790266323169505-30632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790266323169505-30632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790266323169505-30632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:46.619936 3542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:46.620750 3542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:46.621064 3542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:46.639394 3542 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.62:34775
I20260924 16:12:46.639415 3607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.62:34775 every 8 connection(s)
I20260924 16:12:46.640743 3542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
I20260924 16:12:46.644441 3608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:46.647879 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 3542
I20260924 16:12:46.648065 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790266323169505-30632-0/minicluster-data/master-0/wal/instance
I20260924 16:12:46.648923 30632 external_mini_cluster.cc:1428] Setting key 377f85ea9afbd7615e98cdc356e832f1
I20260924 16:12:46.657302 3608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc0369203851432daadd1d5800ff7398: Bootstrap starting.
I20260924 16:12:46.660892 3608 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc0369203851432daadd1d5800ff7398: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:46.662065 3608 log.cc:824] T 00000000000000000000000000000000 P fc0369203851432daadd1d5800ff7398: Log is configured to *not* fsync() on all Append() calls
I20260924 16:12:46.662562 3611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:46.650900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51492 (local address 127.29.234.62:34775)
0924 16:12:46.651876 (+ 976us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:46.651887 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:46.651927 (+ 40us) server_negotiation.cc:407] Connection header received
0924 16:12:46.653548 (+ 1621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:46.653572 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:46.654052 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:46.654397 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:46.654985 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:46.656243 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:46.658031 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:46.658675 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:46.658821 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:46.660086 (+ 1265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:46.660093 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:46.660106 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:12:46.660623 (+ 517us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:46.660708 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:12:46.660729 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:46.660826 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:46.661134 (+ 308us) server_negotiation.cc:299] Negotiation successful
0924 16:12:46.661533 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":362,"threads_started":1}
I20260924 16:12:46.665459 3608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc0369203851432daadd1d5800ff7398: No bootstrap required, opened a new log
I20260924 16:12:46.666697 30632 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:12:46.673832 3608 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc0369203851432daadd1d5800ff7398 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc0369203851432daadd1d5800ff7398" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 34775 } }
I20260924 16:12:46.674265 3608 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc0369203851432daadd1d5800ff7398 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:46.674373 3608 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc0369203851432daadd1d5800ff7398 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc0369203851432daadd1d5800ff7398, State: Initialized, Role: FOLLOWER
I20260924 16:12:46.675028 3608 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc0369203851432daadd1d5800ff7398 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc0369203851432daadd1d5800ff7398" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 34775 } }
I20260924 16:12:46.675311 3608 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc0369203851432daadd1d5800ff7398 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:12:46.675467 3608 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc0369203851432daadd1d5800ff7398 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:12:46.675674 3608 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc0369203851432daadd1d5800ff7398 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:46.678037 3608 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc0369203851432daadd1d5800ff7398 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc0369203851432daadd1d5800ff7398" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 34775 } }
I20260924 16:12:46.678532 3608 leader_election.cc:302] T 00000000000000000000000000000000 P fc0369203851432daadd1d5800ff7398 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc0369203851432daadd1d5800ff7398; no voters:
I20260924 16:12:46.679097 3608 leader_election.cc:288] T 00000000000000000000000000000000 P fc0369203851432daadd1d5800ff7398 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:12:46.679257 3613 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc0369203851432daadd1d5800ff7398 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:12:46.680051 3613 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc0369203851432daadd1d5800ff7398 [term 1 LEADER]: Becoming Leader. State: Replica: fc0369203851432daadd1d5800ff7398, State: Running, Role: LEADER
I20260924 16:12:46.680559 3613 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc0369203851432daadd1d5800ff7398 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc0369203851432daadd1d5800ff7398" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 34775 } }
I20260924 16:12:46.681335 3608 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc0369203851432daadd1d5800ff7398 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:12:46.683444 3614 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc0369203851432daadd1d5800ff7398 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc0369203851432daadd1d5800ff7398" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc0369203851432daadd1d5800ff7398" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 34775 } } }
I20260924 16:12:46.683911 3614 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc0369203851432daadd1d5800ff7398 [sys.catalog]: This master's current role is: LEADER
I20260924 16:12:46.685117 3619 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:12:46.685084 3615 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc0369203851432daadd1d5800ff7398 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc0369203851432daadd1d5800ff7398. Latest consensus state: current_term: 1 leader_uuid: "fc0369203851432daadd1d5800ff7398" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc0369203851432daadd1d5800ff7398" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 34775 } } }
I20260924 16:12:46.685515 3615 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc0369203851432daadd1d5800ff7398 [sys.catalog]: This master's current role is: LEADER
I20260924 16:12:46.690593 3619 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:12:46.699319 3619 catalog_manager.cc:1347] Generated new cluster ID: 67e2452a3b6f45efbe17db5174003a34
I20260924 16:12:46.699441 3619 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:12:46.712419 3619 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:12:46.713837 3619 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:12:46.723097 3619 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc0369203851432daadd1d5800ff7398: Generated new TSK 0
I20260924 16:12:46.723913 3619 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 16:12:46.768505 3639 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260924 16:12:46.768638 3639 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 16:12:46.769819 3611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:46.742473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51508 (local address 127.29.234.62:34775)
0924 16:12:46.742698 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:46.742707 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:46.743080 (+ 373us) server_negotiation.cc:407] Connection header received
0924 16:12:46.743386 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:46.743390 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:46.743466 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:46.743630 (+ 164us) server_negotiation.cc:226] Negotiated authn=JWT
0924 16:12:46.744112 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:46.744845 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:46.746339 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:46.746777 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:46.746912 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:46.747409 (+ 497us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 16:12:46.769031 (+ 21622us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39839/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 16:12:46.769503 (+ 472us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51508: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39839/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":55}
W20260924 16:12:46.769976 3611 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51508: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39839/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260924 16:12:46.770404 3638 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:12:46.742386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:34775 (local address 127.0.0.1:51508)
0924 16:12:46.742973 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:46.742988 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:46.743158 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:46.743647 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:46.743652 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:46.743664 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0924 16:12:46.743945 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:46.743955 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:46.744997 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:46.745007 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:46.746165 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:46.746178 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:46.746956 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:46.746965 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:46.747113 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:46.747147 (+ 34us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 16:12:46.769325 (+ 22178us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39839/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 16:12:46.769657 (+ 332us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.29.234.62:34775: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39839/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":325,"threads_started":1}
I20260924 16:12:46.771860 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 3542
I20260924 16:12:46.773001 3603 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:12:46.866895 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 3542
2026-09-24T16:12:46Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (723 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260924 16:12:46.923342 30632 mini_oidc.cc:152] Starting JWKS server
I20260924 16:12:46.925215 30632 webserver.cc:460] Webserver started at https://127.0.0.1:39397/ using document root <none> and password file <none>
I20260924 16:12:46.925490 30632 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 16:12:46.926120 30632 webserver.cc:460] Webserver started at http://127.0.0.1:38905/ using document root <none> and password file <none>
2026-09-24T16:12:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:12:46Z Disabled control of system clock
I20260924 16:12:46.953302 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790266323169505-30632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790266323169505-30632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790266323169505-30632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:45441
--webserver_interface=127.29.234.62
--webserver_port=0
--builtin_ntp_servers=127.29.234.20:40271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.234.62:45441
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39397/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 16:12:47.274461 3646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:47.274741 3646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:47.274824 3646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:47.281872 3646 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:12:47.281962 3646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:47.281998 3646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:47.282053 3646 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:12:47.282095 3646 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 16:12:47.282153 3646 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 16:12:47.282198 3646 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39397/jwks.json
I20260924 16:12:47.291024 3646 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:40271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790266323169505-30632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790266323169505-30632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.234.62:45441
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:45441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39397/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.234.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790266323169505-30632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:12:47.292299 3646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:47.294010 3646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:47.306133 3651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:47.306361 3652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:47.306625 3654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:47.307302 3646 server_base.cc:1034] running on GCE node
I20260924 16:12:47.308015 3646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:47.309458 3646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:47.310657 3646 hybrid_clock.cc:648] HybridClock initialized: now 1790266367310621 us; error 35 us; skew 500 ppm
I20260924 16:12:47.311017 3646 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 16:12:47.313202 3646 webserver.cc:460] Webserver started at http://127.29.234.62:36473/ using document root <none> and password file <none>
I20260924 16:12:47.313915 3646 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:47.314028 3646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:47.314369 3646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:47.317234 3646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790266323169505-30632-0/minicluster-data/master-0/data/instance:
uuid: "bd4ea4ec237c4b93a9c08056aeac58e0"
format_stamp: "Formatted at 2026-09-24 16:12:47 on dist-test-slave-v5j9"
server_key: "e5d8056a066d8e7ef86b4abbc1aabcf4"
server_key_iv: "3e6d4ed210c26608587d4c00e62d9a0c"
server_key_version: "encryptionkey@0"
I20260924 16:12:47.318104 3646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790266323169505-30632-0/minicluster-data/master-0/wal/instance:
uuid: "bd4ea4ec237c4b93a9c08056aeac58e0"
format_stamp: "Formatted at 2026-09-24 16:12:47 on dist-test-slave-v5j9"
server_key: "e5d8056a066d8e7ef86b4abbc1aabcf4"
server_key_iv: "3e6d4ed210c26608587d4c00e62d9a0c"
server_key_version: "encryptionkey@0"
I20260924 16:12:47.323043 3646 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 16:12:47.326566 3660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:47.327857 3646 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:12:47.328037 3646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790266323169505-30632-0/minicluster-data/master-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790266323169505-30632-0/minicluster-data/master-0/wal
uuid: "bd4ea4ec237c4b93a9c08056aeac58e0"
format_stamp: "Formatted at 2026-09-24 16:12:47 on dist-test-slave-v5j9"
server_key: "e5d8056a066d8e7ef86b4abbc1aabcf4"
server_key_iv: "3e6d4ed210c26608587d4c00e62d9a0c"
server_key_version: "encryptionkey@0"
I20260924 16:12:47.328261 3646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790266323169505-30632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790266323169505-30632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790266323169505-30632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:47.343726 3646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:47.344581 3646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:47.344907 3646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:47.364022 3646 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.62:45441
I20260924 16:12:47.364042 3713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.62:45441 every 8 connection(s)
I20260924 16:12:47.365468 3646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
I20260924 16:12:47.369300 3714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:47.370357 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 3646
I20260924 16:12:47.370572 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790266323169505-30632-0/minicluster-data/master-0/wal/instance
I20260924 16:12:47.371263 30632 external_mini_cluster.cc:1428] Setting key cff22f402c47a454d2416091eb8096de
I20260924 16:12:47.381775 3714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd4ea4ec237c4b93a9c08056aeac58e0: Bootstrap starting.
I20260924 16:12:47.384104 3717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:47.373169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45712 (local address 127.29.234.62:45441)
0924 16:12:47.374212 (+ 1043us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:47.374227 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:47.374274 (+ 47us) server_negotiation.cc:407] Connection header received
0924 16:12:47.375769 (+ 1495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:47.375792 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:47.376279 (+ 487us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:47.376675 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:47.377225 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:47.378505 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:47.379821 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:47.380469 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:47.380658 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:47.381928 (+ 1270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:47.381940 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:47.381957 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:12:47.382493 (+ 536us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:47.382567 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:12:47.382587 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:47.382726 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:47.382937 (+ 211us) server_negotiation.cc:299] Negotiation successful
0924 16:12:47.383243 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":346,"threads_started":1}
I20260924 16:12:47.385175 3714 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd4ea4ec237c4b93a9c08056aeac58e0: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:47.386274 3714 log.cc:824] T 00000000000000000000000000000000 P bd4ea4ec237c4b93a9c08056aeac58e0: Log is configured to *not* fsync() on all Append() calls
I20260924 16:12:47.387954 30632 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:12:47.389101 3714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd4ea4ec237c4b93a9c08056aeac58e0: No bootstrap required, opened a new log
I20260924 16:12:47.396080 3714 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd4ea4ec237c4b93a9c08056aeac58e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd4ea4ec237c4b93a9c08056aeac58e0" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 45441 } }
I20260924 16:12:47.396453 3714 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd4ea4ec237c4b93a9c08056aeac58e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:47.396581 3714 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd4ea4ec237c4b93a9c08056aeac58e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd4ea4ec237c4b93a9c08056aeac58e0, State: Initialized, Role: FOLLOWER
I20260924 16:12:47.397222 3714 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd4ea4ec237c4b93a9c08056aeac58e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd4ea4ec237c4b93a9c08056aeac58e0" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 45441 } }
I20260924 16:12:47.397513 3714 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd4ea4ec237c4b93a9c08056aeac58e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:12:47.397660 3714 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd4ea4ec237c4b93a9c08056aeac58e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:12:47.397843 3714 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd4ea4ec237c4b93a9c08056aeac58e0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:47.399905 3714 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd4ea4ec237c4b93a9c08056aeac58e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd4ea4ec237c4b93a9c08056aeac58e0" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 45441 } }
I20260924 16:12:47.400437 3714 leader_election.cc:302] T 00000000000000000000000000000000 P bd4ea4ec237c4b93a9c08056aeac58e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd4ea4ec237c4b93a9c08056aeac58e0; no voters:
I20260924 16:12:47.401005 3714 leader_election.cc:288] T 00000000000000000000000000000000 P bd4ea4ec237c4b93a9c08056aeac58e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:12:47.401147 3719 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd4ea4ec237c4b93a9c08056aeac58e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:12:47.401887 3719 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd4ea4ec237c4b93a9c08056aeac58e0 [term 1 LEADER]: Becoming Leader. State: Replica: bd4ea4ec237c4b93a9c08056aeac58e0, State: Running, Role: LEADER
I20260924 16:12:47.402375 3719 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd4ea4ec237c4b93a9c08056aeac58e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd4ea4ec237c4b93a9c08056aeac58e0" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 45441 } }
I20260924 16:12:47.403151 3714 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd4ea4ec237c4b93a9c08056aeac58e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:12:47.405365 3721 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd4ea4ec237c4b93a9c08056aeac58e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd4ea4ec237c4b93a9c08056aeac58e0. Latest consensus state: current_term: 1 leader_uuid: "bd4ea4ec237c4b93a9c08056aeac58e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd4ea4ec237c4b93a9c08056aeac58e0" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 45441 } } }
I20260924 16:12:47.405738 3721 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd4ea4ec237c4b93a9c08056aeac58e0 [sys.catalog]: This master's current role is: LEADER
I20260924 16:12:47.407473 3726 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:12:47.407334 3720 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd4ea4ec237c4b93a9c08056aeac58e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd4ea4ec237c4b93a9c08056aeac58e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd4ea4ec237c4b93a9c08056aeac58e0" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 45441 } } }
I20260924 16:12:47.407655 3720 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd4ea4ec237c4b93a9c08056aeac58e0 [sys.catalog]: This master's current role is: LEADER
I20260924 16:12:47.412611 3726 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:12:47.421636 3726 catalog_manager.cc:1347] Generated new cluster ID: b4a94cf208484dc186e7fa3d9443eac1
I20260924 16:12:47.421756 3726 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:12:47.433269 3726 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:12:47.435002 3726 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:12:47.451932 3726 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd4ea4ec237c4b93a9c08056aeac58e0: Generated new TSK 0
I20260924 16:12:47.452883 3726 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:12:47.484671 3717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:47.476227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45728 (local address 127.29.234.62:45441)
0924 16:12:47.476463 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:47.476471 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:47.476817 (+ 346us) server_negotiation.cc:407] Connection header received
0924 16:12:47.477033 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:47.477038 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:47.477134 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:47.477292 (+ 158us) server_negotiation.cc:226] Negotiated authn=JWT
0924 16:12:47.477778 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:47.478942 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:47.480264 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:47.480968 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:47.481090 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:47.481646 (+ 556us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 16:12:47.484025 (+ 2379us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39397/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 16:12:47.484355 (+ 330us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45728: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39397/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":86}
W20260924 16:12:47.484814 3717 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45728: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39397/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260924 16:12:47.486410 3743 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:12:47.476118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:45441 (local address 127.0.0.1:45728)
0924 16:12:47.476717 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:47.476737 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:47.476861 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:47.477305 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:47.477309 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:47.477318 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0924 16:12:47.477618 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:47.477627 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:47.479072 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:47.479075 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:47.480127 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:47.480137 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:47.481134 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:47.481139 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:47.481264 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:47.481294 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 16:12:47.484235 (+ 2941us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39397/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 16:12:47.485971 (+ 1736us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.29.234.62:45441: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39397/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":467,"mutex_wait_us":1433,"thread_start_us":314,"threads_started":1}
I20260924 16:12:47.487409 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 3646
I20260924 16:12:47.488546 3709 generic_service.cc:196] Checking for leaks (request via RPC)
E20260924 16:12:47.496488 3744 webserver.cc:533] Webserver: sslize failed:
E20260924 16:12:47.496610 3744 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 16:12:47.580876 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 3646
2026-09-24T16:12:47Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (713 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-24T16:12:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:12:47Z Disabled control of system clock
I20260924 16:12:47.664261 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790266323169505-30632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790266323169505-30632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790266323169505-30632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:35181
--webserver_interface=127.29.234.62
--webserver_port=0
--builtin_ntp_servers=127.29.234.20:45119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.234.62:35181
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790266323169505-30632-0/testchainca.cert with env {}
W20260924 16:12:47.998571 3749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:47.998858 3749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:47.998941 3749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:48.005947 3749 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:12:48.006047 3749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:48.006084 3749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:48.006166 3749 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:12:48.006212 3749 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 16:12:48.006268 3749 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 16:12:48.006307 3749 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260924 16:12:48.015919 3749 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:45119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790266323169505-30632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790266323169505-30632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.234.62:35181
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:35181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.234.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790266323169505-30632-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790266323169505-30632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:12:48.017287 3749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:48.018704 3749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:48.030788 3755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:48.030874 3754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:48.031328 3757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:48.031433 3749 server_base.cc:1034] running on GCE node
I20260924 16:12:48.032385 3749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:48.033890 3749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:48.035121 3749 hybrid_clock.cc:648] HybridClock initialized: now 1790266368035079 us; error 52 us; skew 500 ppm
I20260924 16:12:48.035483 3749 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 16:12:48.037835 3749 webserver.cc:460] Webserver started at http://127.29.234.62:43849/ using document root <none> and password file <none>
I20260924 16:12:48.038535 3749 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:48.038669 3749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:48.039006 3749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:48.041906 3749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790266323169505-30632-0/minicluster-data/master-0/data/instance:
uuid: "18d6121a812a47b1818049863f1fecc6"
format_stamp: "Formatted at 2026-09-24 16:12:48 on dist-test-slave-v5j9"
server_key: "b57a9f2eb8fe39797e79d97a6eccd576"
server_key_iv: "8d9a76dcb559f953fe6c8bcda11063a3"
server_key_version: "encryptionkey@0"
I20260924 16:12:48.042693 3749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790266323169505-30632-0/minicluster-data/master-0/wal/instance:
uuid: "18d6121a812a47b1818049863f1fecc6"
format_stamp: "Formatted at 2026-09-24 16:12:48 on dist-test-slave-v5j9"
server_key: "b57a9f2eb8fe39797e79d97a6eccd576"
server_key_iv: "8d9a76dcb559f953fe6c8bcda11063a3"
server_key_version: "encryptionkey@0"
I20260924 16:12:48.047662 3749 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260924 16:12:48.050987 3763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:48.052433 3749 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 16:12:48.052656 3749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790266323169505-30632-0/minicluster-data/master-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790266323169505-30632-0/minicluster-data/master-0/wal
uuid: "18d6121a812a47b1818049863f1fecc6"
format_stamp: "Formatted at 2026-09-24 16:12:48 on dist-test-slave-v5j9"
server_key: "b57a9f2eb8fe39797e79d97a6eccd576"
server_key_iv: "8d9a76dcb559f953fe6c8bcda11063a3"
server_key_version: "encryptionkey@0"
I20260924 16:12:48.052901 3749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790266323169505-30632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790266323169505-30632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790266323169505-30632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:48.079627 3749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:48.080502 3749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:48.080806 3749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:48.099154 3749 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.62:35181
I20260924 16:12:48.099169 3814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.62:35181 every 8 connection(s)
I20260924 16:12:48.100486 3749 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
I20260924 16:12:48.103192 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 3749
I20260924 16:12:48.103404 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790266323169505-30632-0/minicluster-data/master-0/wal/instance
I20260924 16:12:48.104040 30632 external_mini_cluster.cc:1428] Setting key 9f50b50492d413535453f35044e6ff5c
I20260924 16:12:48.104423 3815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:48.117419 3819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:48.105762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54216 (local address 127.29.234.62:35181)
0924 16:12:48.106712 (+ 950us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:48.106729 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:48.106769 (+ 40us) server_negotiation.cc:407] Connection header received
0924 16:12:48.108242 (+ 1473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:48.108263 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:48.108717 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:48.109044 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:48.109668 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:48.110989 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:48.112822 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:48.113512 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:48.113663 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:48.114924 (+ 1261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:48.114932 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:48.114948 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:12:48.115483 (+ 535us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:48.115555 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:12:48.115579 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:48.115663 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:48.115900 (+ 237us) server_negotiation.cc:299] Negotiation successful
0924 16:12:48.116230 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":357,"threads_started":1}
I20260924 16:12:48.118513 3815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18d6121a812a47b1818049863f1fecc6: Bootstrap starting.
I20260924 16:12:48.121106 30632 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:12:48.121855 3815 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 18d6121a812a47b1818049863f1fecc6: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:48.122830 3815 log.cc:824] T 00000000000000000000000000000000 P 18d6121a812a47b1818049863f1fecc6: Log is configured to *not* fsync() on all Append() calls
I20260924 16:12:48.125922 3815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18d6121a812a47b1818049863f1fecc6: No bootstrap required, opened a new log
I20260924 16:12:48.133126 3815 raft_consensus.cc:348] T 00000000000000000000000000000000 P 18d6121a812a47b1818049863f1fecc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18d6121a812a47b1818049863f1fecc6" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 35181 } }
I20260924 16:12:48.133497 3815 raft_consensus.cc:374] T 00000000000000000000000000000000 P 18d6121a812a47b1818049863f1fecc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:48.133600 3815 raft_consensus.cc:729] T 00000000000000000000000000000000 P 18d6121a812a47b1818049863f1fecc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18d6121a812a47b1818049863f1fecc6, State: Initialized, Role: FOLLOWER
I20260924 16:12:48.134315 3815 consensus_queue.cc:260] T 00000000000000000000000000000000 P 18d6121a812a47b1818049863f1fecc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18d6121a812a47b1818049863f1fecc6" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 35181 } }
I20260924 16:12:48.134579 3815 raft_consensus.cc:388] T 00000000000000000000000000000000 P 18d6121a812a47b1818049863f1fecc6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:12:48.134708 3815 raft_consensus.cc:482] T 00000000000000000000000000000000 P 18d6121a812a47b1818049863f1fecc6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:12:48.134919 3815 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 18d6121a812a47b1818049863f1fecc6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:48.137183 3815 raft_consensus.cc:504] T 00000000000000000000000000000000 P 18d6121a812a47b1818049863f1fecc6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18d6121a812a47b1818049863f1fecc6" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 35181 } }
I20260924 16:12:48.137738 3815 leader_election.cc:302] T 00000000000000000000000000000000 P 18d6121a812a47b1818049863f1fecc6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 18d6121a812a47b1818049863f1fecc6; no voters:
I20260924 16:12:48.138338 3815 leader_election.cc:288] T 00000000000000000000000000000000 P 18d6121a812a47b1818049863f1fecc6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:12:48.138511 3821 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 18d6121a812a47b1818049863f1fecc6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:12:48.139184 3821 raft_consensus.cc:686] T 00000000000000000000000000000000 P 18d6121a812a47b1818049863f1fecc6 [term 1 LEADER]: Becoming Leader. State: Replica: 18d6121a812a47b1818049863f1fecc6, State: Running, Role: LEADER
I20260924 16:12:48.139698 3821 consensus_queue.cc:237] T 00000000000000000000000000000000 P 18d6121a812a47b1818049863f1fecc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18d6121a812a47b1818049863f1fecc6" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 35181 } }
I20260924 16:12:48.140614 3815 sys_catalog.cc:564] T 00000000000000000000000000000000 P 18d6121a812a47b1818049863f1fecc6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:12:48.142678 3823 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18d6121a812a47b1818049863f1fecc6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 18d6121a812a47b1818049863f1fecc6. Latest consensus state: current_term: 1 leader_uuid: "18d6121a812a47b1818049863f1fecc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18d6121a812a47b1818049863f1fecc6" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 35181 } } }
I20260924 16:12:48.143087 3823 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18d6121a812a47b1818049863f1fecc6 [sys.catalog]: This master's current role is: LEADER
I20260924 16:12:48.143705 3822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18d6121a812a47b1818049863f1fecc6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "18d6121a812a47b1818049863f1fecc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18d6121a812a47b1818049863f1fecc6" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 35181 } } }
I20260924 16:12:48.144094 3822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18d6121a812a47b1818049863f1fecc6 [sys.catalog]: This master's current role is: LEADER
I20260924 16:12:48.146512 3830 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:12:48.152251 3830 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:12:48.160194 3830 catalog_manager.cc:1347] Generated new cluster ID: 18c2bc445cd6420eab089fc0b455b2fc
I20260924 16:12:48.160305 3830 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:12:48.170079 3830 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:12:48.171609 3830 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:12:48.178239 3830 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 18d6121a812a47b1818049863f1fecc6: Generated new TSK 0
I20260924 16:12:48.179014 3830 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:12:48.615852 3819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:48.190477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54222 (local address 127.29.234.62:35181)
0924 16:12:48.190698 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:48.190705 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:48.191056 (+ 351us) server_negotiation.cc:407] Connection header received
0924 16:12:48.191324 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:48.191329 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:48.191396 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:48.191570 (+ 174us) server_negotiation.cc:226] Negotiated authn=JWT
0924 16:12:48.192116 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:48.192850 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:48.194345 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:48.194813 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:48.194957 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:48.195462 (+ 505us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 16:12:48.615037 (+419575us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 16:12:48.615545 (+ 508us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54222: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":51}
W20260924 16:12:48.616012 3819 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54222: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260924 16:12:48.616145 3847 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:12:48.190348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:35181 (local address 127.0.0.1:54222)
0924 16:12:48.190950 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:48.190965 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:48.191132 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:48.191581 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:48.191586 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:48.191599 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0924 16:12:48.191900 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:48.191929 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:48.193020 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:48.193025 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:48.194191 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:48.194203 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:48.194993 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:48.194999 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:48.195145 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:48.195180 (+ 35us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 16:12:48.615385 (+420205us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 16:12:48.615699 (+ 314us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.29.234.62:35181: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":348,"threads_started":1}
I20260924 16:12:48.617863 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 3749
I20260924 16:12:48.619033 3810 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:12:48.722461 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 3749
2026-09-24T16:12:48Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1123 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (409 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (412 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:12:49 dist-test-slave-v5j9 krb5kdc[3862](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:12:49 dist-test-slave-v5j9 krb5kdc[3862](info): set up 2 sockets
Sep 24 16:12:49 dist-test-slave-v5j9 krb5kdc[3862](info): commencing operation
krb5kdc: starting...
W20260924 16:12:50.656241 30632 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.001s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Sep 24 16:12:50 dist-test-slave-v5j9 krb5kdc[3862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:12:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:12:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.234.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab.
Entry for principal kudu/127.29.234.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab.
I20260924 16:12:50.873721 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:35867
--webserver_interface=127.29.234.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab
--principal=kudu/127.29.234.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:33045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.234.62:35867
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:12:51.203389 3875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:51.203766 3875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:51.203948 3875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:51.211239 3875 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:12:51.211416 3875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:51.211529 3875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:51.211638 3875 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:12:51.211730 3875 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:12:51.221827 3875 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:33045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.234.62:35867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab
--principal=kudu/127.29.234.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:35867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.234.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:12:51.223278 3875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:51.224676 3875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:51.237279 3881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:51.237433 3880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:51.237430 3883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:51.238401 3875 server_base.cc:1034] running on GCE node
I20260924 16:12:51.239202 3875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:51.240743 3875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:51.241952 3875 hybrid_clock.cc:648] HybridClock initialized: now 1790266371241907 us; error 29 us; skew 500 ppm
I20260924 16:12:51.242336 3875 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:12:51 dist-test-slave-v5j9 krb5kdc[3862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266371, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:51.245574 3875 init.cc:377] Logged in from keytab as kudu/127.29.234.62@KRBTEST.COM (short username kudu)
I20260924 16:12:51.247184 3875 webserver.cc:460] Webserver started at http://127.29.234.62:41719/ using document root <none> and password file <none>
I20260924 16:12:51.247916 3875 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:51.248066 3875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:51.248414 3875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:51.251590 3875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/minicluster-data/master-0/data/instance:
uuid: "ad01893986f74744890c45528990bb58"
format_stamp: "Formatted at 2026-09-24 16:12:51 on dist-test-slave-v5j9"
server_key: "07f90df2f2856301478eba8ea8db471c"
server_key_iv: "cacec1bb94dba844542fcb7d1d83798d"
server_key_version: "encryptionkey@0"
I20260924 16:12:51.252422 3875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/minicluster-data/master-0/wal/instance:
uuid: "ad01893986f74744890c45528990bb58"
format_stamp: "Formatted at 2026-09-24 16:12:51 on dist-test-slave-v5j9"
server_key: "07f90df2f2856301478eba8ea8db471c"
server_key_iv: "cacec1bb94dba844542fcb7d1d83798d"
server_key_version: "encryptionkey@0"
I20260924 16:12:51.257853 3875 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 16:12:51.261408 3890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:51.262740 3875 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:12:51.262933 3875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/minicluster-data/master-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/minicluster-data/master-0/wal
uuid: "ad01893986f74744890c45528990bb58"
format_stamp: "Formatted at 2026-09-24 16:12:51 on dist-test-slave-v5j9"
server_key: "07f90df2f2856301478eba8ea8db471c"
server_key_iv: "cacec1bb94dba844542fcb7d1d83798d"
server_key_version: "encryptionkey@0"
I20260924 16:12:51.263125 3875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:51.287727 3875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:51.292209 3875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:51.292512 3875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:51.311797 3875 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.62:35867
I20260924 16:12:51.311837 3942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.62:35867 every 8 connection(s)
I20260924 16:12:51.313253 3875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
I20260924 16:12:51.314827 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 3875
I20260924 16:12:51.315066 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/minicluster-data/master-0/wal/instance
I20260924 16:12:51.315883 30632 external_mini_cluster.cc:1428] Setting key 2dd327d8d8af492b6da490a482f16d36
I20260924 16:12:51.318415 3943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 16:12:51 dist-test-slave-v5j9 krb5kdc[3862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:12:51.332118 3943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad01893986f74744890c45528990bb58: Bootstrap starting.
I20260924 16:12:51.336346 3943 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad01893986f74744890c45528990bb58: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:51.337530 3943 log.cc:824] T 00000000000000000000000000000000 P ad01893986f74744890c45528990bb58: Log is configured to *not* fsync() on all Append() calls
I20260924 16:12:51.340107 3946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:51.318067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42570 (local address 127.29.234.62:35867)
0924 16:12:51.318885 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:51.318897 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:51.318939 (+ 42us) server_negotiation.cc:407] Connection header received
0924 16:12:51.320605 (+ 1666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:51.320628 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:51.321023 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:51.321441 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:51.323146 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:51.324479 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:51.326290 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:51.326959 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:51.327091 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:51.331794 (+ 4703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:51.331805 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:51.331828 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:51.331879 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:51.335831 (+ 3952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:51.336529 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:51.336537 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:51.336543 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:51.336718 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:51.337157 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:51.337164 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:51.337172 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:51.337840 (+ 668us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:51.338153 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:51.338632 (+ 479us) server_negotiation.cc:299] Negotiation successful
0924 16:12:51.339120 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":392,"threads_started":1}
I20260924 16:12:51.340512 3943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad01893986f74744890c45528990bb58: No bootstrap required, opened a new log
I20260924 16:12:51.348064 3943 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad01893986f74744890c45528990bb58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad01893986f74744890c45528990bb58" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 35867 } }
I20260924 16:12:51.348441 3943 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad01893986f74744890c45528990bb58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:51.348548 3943 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad01893986f74744890c45528990bb58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad01893986f74744890c45528990bb58, State: Initialized, Role: FOLLOWER
I20260924 16:12:51.349087 3943 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad01893986f74744890c45528990bb58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad01893986f74744890c45528990bb58" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 35867 } }
I20260924 16:12:51.349328 3943 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad01893986f74744890c45528990bb58 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:12:51.349541 3943 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad01893986f74744890c45528990bb58 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:12:51.349737 3943 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad01893986f74744890c45528990bb58 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:51.351855 3943 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad01893986f74744890c45528990bb58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad01893986f74744890c45528990bb58" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 35867 } }
I20260924 16:12:51.352386 3943 leader_election.cc:302] T 00000000000000000000000000000000 P ad01893986f74744890c45528990bb58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad01893986f74744890c45528990bb58; no voters:
I20260924 16:12:51.353010 3943 leader_election.cc:288] T 00000000000000000000000000000000 P ad01893986f74744890c45528990bb58 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:12:51.353138 3948 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad01893986f74744890c45528990bb58 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:12:51.353842 3948 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad01893986f74744890c45528990bb58 [term 1 LEADER]: Becoming Leader. State: Replica: ad01893986f74744890c45528990bb58, State: Running, Role: LEADER
I20260924 16:12:51.354379 3948 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad01893986f74744890c45528990bb58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad01893986f74744890c45528990bb58" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 35867 } }
I20260924 16:12:51.355180 3943 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad01893986f74744890c45528990bb58 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:12:51.357023 3949 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad01893986f74744890c45528990bb58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad01893986f74744890c45528990bb58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad01893986f74744890c45528990bb58" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 35867 } } }
I20260924 16:12:51.357497 3949 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad01893986f74744890c45528990bb58 [sys.catalog]: This master's current role is: LEADER
I20260924 16:12:51.357012 3950 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad01893986f74744890c45528990bb58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad01893986f74744890c45528990bb58. Latest consensus state: current_term: 1 leader_uuid: "ad01893986f74744890c45528990bb58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad01893986f74744890c45528990bb58" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 35867 } } }
I20260924 16:12:51.357828 3950 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad01893986f74744890c45528990bb58 [sys.catalog]: This master's current role is: LEADER
I20260924 16:12:51.359817 3954 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:12:51.365510 3954 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:12:51.374956 3954 catalog_manager.cc:1347] Generated new cluster ID: 7e04e0dada734584af8ba2a03638f2ff
I20260924 16:12:51.375098 3954 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:12:51.388185 3954 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:12:51.389653 3954 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:12:51.414352 3954 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad01893986f74744890c45528990bb58: Generated new TSK 0
I20260924 16:12:51.415560 3954 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.234.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab.
Entry for principal kudu/127.29.234.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab.
I20260924 16:12:51.469571 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.1:0
--local_ip_for_outbound_sockets=127.29.234.1
--webserver_interface=127.29.234.1
--webserver_port=0
--tserver_master_addrs=127.29.234.62:35867
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab
--principal=kudu/127.29.234.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:33045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:12:51.797413 3969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:51.797784 3969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:51.797966 3969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:51.805212 3969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:51.805416 3969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:51.805620 3969 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.1
I20260924 16:12:51.815407 3969 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:33045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab
--principal=kudu/127.29.234.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.234.1
--webserver_port=0
--tserver_master_addrs=127.29.234.62:35867
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.1
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:12:51.816848 3969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:51.818288 3969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:51.831797 3974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:51.831838 3977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:51.831842 3975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:51.832947 3969 server_base.cc:1034] running on GCE node
I20260924 16:12:51.833643 3969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:51.834653 3969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:51.835848 3969 hybrid_clock.cc:648] HybridClock initialized: now 1790266371835800 us; error 45 us; skew 500 ppm
I20260924 16:12:51.836171 3969 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:12:51 dist-test-slave-v5j9 krb5kdc[3862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266371, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:51.851608 3969 init.cc:377] Logged in from keytab as kudu/127.29.234.1@KRBTEST.COM (short username kudu)
I20260924 16:12:51.853236 3969 webserver.cc:460] Webserver started at http://127.29.234.1:40207/ using document root <none> and password file <none>
I20260924 16:12:51.854012 3969 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:51.854162 3969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:51.854492 3969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:51.857896 3969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/minicluster-data/ts-0/data/instance:
uuid: "67933ebed2c5421f8647e283772ed10e"
format_stamp: "Formatted at 2026-09-24 16:12:51 on dist-test-slave-v5j9"
server_key: "5afe212f12050d7ce966526a4ffa6791"
server_key_iv: "9722485c9432db1360ab6b621b15e249"
server_key_version: "encryptionkey@0"
I20260924 16:12:51.858729 3969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/minicluster-data/ts-0/wal/instance:
uuid: "67933ebed2c5421f8647e283772ed10e"
format_stamp: "Formatted at 2026-09-24 16:12:51 on dist-test-slave-v5j9"
server_key: "5afe212f12050d7ce966526a4ffa6791"
server_key_iv: "9722485c9432db1360ab6b621b15e249"
server_key_version: "encryptionkey@0"
I20260924 16:12:51.864152 3969 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260924 16:12:51.867717 3984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:51.869123 3969 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 16:12:51.869292 3969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/minicluster-data/ts-0/wal
uuid: "67933ebed2c5421f8647e283772ed10e"
format_stamp: "Formatted at 2026-09-24 16:12:51 on dist-test-slave-v5j9"
server_key: "5afe212f12050d7ce966526a4ffa6791"
server_key_iv: "9722485c9432db1360ab6b621b15e249"
server_key_version: "encryptionkey@0"
I20260924 16:12:51.869553 3969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:51.903326 3969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:51.907768 3969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:51.908066 3969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:51.909049 3969 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:12:51.910401 3969 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:12:51.910473 3969 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:51.910609 3969 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:12:51.910671 3969 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:51.943778 3969 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.1:32953
I20260924 16:12:51.943878 4099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.1:32953 every 8 connection(s)
I20260924 16:12:51.945225 3969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
I20260924 16:12:51.951897 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 3969
I20260924 16:12:51.952105 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/minicluster-data/ts-0/wal/instance
I20260924 16:12:51.952888 30632 external_mini_cluster.cc:1428] Setting key 70d40b05382f2756c34c784065d04dbb
Sep 24 16:12:51 dist-test-slave-v5j9 krb5kdc[3862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266371, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.1@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:12:51.966754 4102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:51.947747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.1:39717 (local address 127.29.234.62:35867)
0924 16:12:51.948208 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:51.948212 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:51.949015 (+ 803us) server_negotiation.cc:407] Connection header received
0924 16:12:51.950898 (+ 1883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:51.950903 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:51.950997 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:51.951113 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:51.953237 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:51.954172 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:51.956425 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:51.956922 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:51.957022 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:51.961190 (+ 4168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:51.961196 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:51.961199 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:51.961229 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:51.963690 (+ 2461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:51.964387 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:51.964395 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:51.964400 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:51.964527 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:51.964948 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:51.964954 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:51.964957 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:51.965220 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:51.965406 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:51.966110 (+ 704us) server_negotiation.cc:299] Negotiation successful
0924 16:12:51.966384 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":245,"threads_started":1}
I20260924 16:12:51.967860 4103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:51.948177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:35867 (local address 127.29.234.1:39717)
0924 16:12:51.948846 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:51.948883 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:51.950617 (+ 1734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:51.951394 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:51.951409 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:51.952190 (+ 781us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:51.953048 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:51.953077 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:51.954333 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:51.954339 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:51.956274 (+ 1935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:51.956292 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:51.957162 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:51.957171 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:51.957528 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:51.958246 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:51.958285 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:51.960959 (+ 2674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:51.963830 (+ 2871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:51.963839 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:51.963855 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:51.964256 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:51.964636 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:51.964641 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:51.964645 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:51.964833 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:51.965381 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:51.965390 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:51.965737 (+ 347us) client_negotiation.cc:769] Sending connection context
0924 16:12:51.966020 (+ 283us) client_negotiation.cc:240] Negotiation successful
0924 16:12:51.966604 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":266,"threads_started":1}
I20260924 16:12:51.969606 4100 heartbeater.cc:347] Connected to a master server at 127.29.234.62:35867
I20260924 16:12:51.969969 4100 heartbeater.cc:464] Registering TS with master...
I20260924 16:12:51.970978 4100 heartbeater.cc:511] Master 127.29.234.62:35867 requested a full tablet report, sending...
I20260924 16:12:51.973708 3907 ts_manager.cc:194] Registered new tserver with Master: 67933ebed2c5421f8647e283772ed10e (127.29.234.1:32953)
WARNING: no policy specified for kudu/127.29.234.2@KRBTEST.COM; defaulting to no policy
I20260924 16:12:51.975350 3907 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.234.1@KRBTEST.COM'} at 127.29.234.1:39717
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab.
Entry for principal kudu/127.29.234.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab.
I20260924 16:12:52.004441 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.2:0
--local_ip_for_outbound_sockets=127.29.234.2
--webserver_interface=127.29.234.2
--webserver_port=0
--tserver_master_addrs=127.29.234.62:35867
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab
--principal=kudu/127.29.234.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:33045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:12:52.339444 4106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:52.339749 4106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:52.339845 4106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:52.347577 4106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:52.347704 4106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:52.347823 4106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.2
I20260924 16:12:52.358032 4106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:33045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab
--principal=kudu/127.29.234.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.234.2
--webserver_port=0
--tserver_master_addrs=127.29.234.62:35867
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.2
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:12:52.359450 4106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:52.360949 4106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:52.375325 4112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:52.375371 4113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:52.375603 4115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:52.376171 4106 server_base.cc:1034] running on GCE node
I20260924 16:12:52.376830 4106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:52.377942 4106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:52.379172 4106 hybrid_clock.cc:648] HybridClock initialized: now 1790266372379128 us; error 44 us; skew 500 ppm
I20260924 16:12:52.379525 4106 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:12:52 dist-test-slave-v5j9 krb5kdc[3862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266372, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:52.382894 4106 init.cc:377] Logged in from keytab as kudu/127.29.234.2@KRBTEST.COM (short username kudu)
I20260924 16:12:52.384405 4106 webserver.cc:460] Webserver started at http://127.29.234.2:42539/ using document root <none> and password file <none>
I20260924 16:12:52.385128 4106 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:52.385288 4106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:52.385635 4106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:52.388603 4106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/minicluster-data/ts-1/data/instance:
uuid: "acb186238e4d4e3faeac181dc493cef5"
format_stamp: "Formatted at 2026-09-24 16:12:52 on dist-test-slave-v5j9"
server_key: "e491a3f7b1f346b3967f3428029c9098"
server_key_iv: "27f254492d7dcd596e1b97fad2a6f7e4"
server_key_version: "encryptionkey@0"
I20260924 16:12:52.389449 4106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/minicluster-data/ts-1/wal/instance:
uuid: "acb186238e4d4e3faeac181dc493cef5"
format_stamp: "Formatted at 2026-09-24 16:12:52 on dist-test-slave-v5j9"
server_key: "e491a3f7b1f346b3967f3428029c9098"
server_key_iv: "27f254492d7dcd596e1b97fad2a6f7e4"
server_key_version: "encryptionkey@0"
I20260924 16:12:52.394881 4106 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260924 16:12:52.398602 4122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:52.400029 4106 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 16:12:52.400219 4106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/minicluster-data/ts-1/wal
uuid: "acb186238e4d4e3faeac181dc493cef5"
format_stamp: "Formatted at 2026-09-24 16:12:52 on dist-test-slave-v5j9"
server_key: "e491a3f7b1f346b3967f3428029c9098"
server_key_iv: "27f254492d7dcd596e1b97fad2a6f7e4"
server_key_version: "encryptionkey@0"
I20260924 16:12:52.400506 4106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:52.417725 4106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:52.422307 4106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:52.422603 4106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:52.423624 4106 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:12:52.424978 4106 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:12:52.425084 4106 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:52.425251 4106 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:12:52.425326 4106 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:52.459147 4106 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.2:46711
I20260924 16:12:52.459195 4236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.2:46711 every 8 connection(s)
I20260924 16:12:52.460471 4106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
I20260924 16:12:52.466212 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 4106
I20260924 16:12:52.466389 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/minicluster-data/ts-1/wal/instance
I20260924 16:12:52.466936 30632 external_mini_cluster.cc:1428] Setting key cebb89dd9bd96c99bc551e0228b6bab2
Sep 24 16:12:52 dist-test-slave-v5j9 krb5kdc[3862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266372, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.2@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:12:52.481742 4102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:52.463101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.2:57283 (local address 127.29.234.62:35867)
0924 16:12:52.463284 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:52.463289 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:52.464499 (+ 1210us) server_negotiation.cc:407] Connection header received
0924 16:12:52.466429 (+ 1930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:52.466438 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:52.466558 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:52.466668 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:52.468533 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:52.469632 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:52.471164 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:52.471908 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:52.472056 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:52.475901 (+ 3845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:52.475906 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:52.475910 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:52.475941 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:52.478528 (+ 2587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:52.479453 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:52.479456 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:52.479459 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:52.479576 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:52.480027 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:52.480030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:52.480032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:52.480261 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:52.480370 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:52.481241 (+ 871us) server_negotiation.cc:299] Negotiation successful
0924 16:12:52.481467 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":67}
I20260924 16:12:52.483278 4239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:52.463637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:35867 (local address 127.29.234.2:57283)
0924 16:12:52.464340 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:52.464375 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:52.466100 (+ 1725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:52.466919 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:52.466927 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:52.467552 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:52.468343 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:52.468369 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:52.469779 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:52.469783 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:52.471015 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:52.471028 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:52.472198 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:52.472208 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:52.472440 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:52.473213 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:52.473236 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:52.475669 (+ 2433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:52.478672 (+ 3003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:52.478683 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:52.478703 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:52.479316 (+ 613us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:52.479679 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:52.479686 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:52.479693 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:52.479924 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:52.480412 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:52.480423 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:52.480877 (+ 454us) client_negotiation.cc:769] Sending connection context
0924 16:12:52.481182 (+ 305us) client_negotiation.cc:240] Negotiation successful
0924 16:12:52.481813 (+ 631us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":273,"threads_started":1}
I20260924 16:12:52.484863 4237 heartbeater.cc:347] Connected to a master server at 127.29.234.62:35867
I20260924 16:12:52.485203 4237 heartbeater.cc:464] Registering TS with master...
I20260924 16:12:52.486229 4237 heartbeater.cc:511] Master 127.29.234.62:35867 requested a full tablet report, sending...
I20260924 16:12:52.487974 3907 ts_manager.cc:194] Registered new tserver with Master: acb186238e4d4e3faeac181dc493cef5 (127.29.234.2:46711)
WARNING: no policy specified for kudu/127.29.234.3@KRBTEST.COM; defaulting to no policy
I20260924 16:12:52.489207 3907 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.234.2@KRBTEST.COM'} at 127.29.234.2:57283
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab.
Entry for principal kudu/127.29.234.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab.
I20260924 16:12:52.516659 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.3:0
--local_ip_for_outbound_sockets=127.29.234.3
--webserver_interface=127.29.234.3
--webserver_port=0
--tserver_master_addrs=127.29.234.62:35867
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab
--principal=kudu/127.29.234.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:33045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:12:52.842507 4242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:52.842772 4242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:52.842900 4242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:52.850245 4242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:52.850370 4242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:52.850553 4242 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.3
I20260924 16:12:52.860292 4242 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:33045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab
--principal=kudu/127.29.234.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.234.3
--webserver_port=0
--tserver_master_addrs=127.29.234.62:35867
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.3
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:12:52.861661 4242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:52.862969 4242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:52.876194 4247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:52.876314 4248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:52.876335 4250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:52.877166 4242 server_base.cc:1034] running on GCE node
I20260924 16:12:52.877797 4242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:52.878672 4242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:52.879895 4242 hybrid_clock.cc:648] HybridClock initialized: now 1790266372879850 us; error 38 us; skew 500 ppm
I20260924 16:12:52.880206 4242 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:12:52 dist-test-slave-v5j9 krb5kdc[3862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266372, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:52.883262 4242 init.cc:377] Logged in from keytab as kudu/127.29.234.3@KRBTEST.COM (short username kudu)
I20260924 16:12:52.884609 4242 webserver.cc:460] Webserver started at http://127.29.234.3:36989/ using document root <none> and password file <none>
I20260924 16:12:52.885409 4242 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:52.885555 4242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:52.885869 4242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:52.888708 4242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/minicluster-data/ts-2/data/instance:
uuid: "0851b8f78bc74f27b871f6530ac6ae3e"
format_stamp: "Formatted at 2026-09-24 16:12:52 on dist-test-slave-v5j9"
server_key: "ace1c4d91dc14cb8944e1ab8887fd2fd"
server_key_iv: "e5bc5afb9672fecd48d9b5218483627f"
server_key_version: "encryptionkey@0"
I20260924 16:12:52.889624 4242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/minicluster-data/ts-2/wal/instance:
uuid: "0851b8f78bc74f27b871f6530ac6ae3e"
format_stamp: "Formatted at 2026-09-24 16:12:52 on dist-test-slave-v5j9"
server_key: "ace1c4d91dc14cb8944e1ab8887fd2fd"
server_key_iv: "e5bc5afb9672fecd48d9b5218483627f"
server_key_version: "encryptionkey@0"
I20260924 16:12:52.894819 4242 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260924 16:12:52.898396 4257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:52.899765 4242 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 16:12:52.899943 4242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/minicluster-data/ts-2/wal
uuid: "0851b8f78bc74f27b871f6530ac6ae3e"
format_stamp: "Formatted at 2026-09-24 16:12:52 on dist-test-slave-v5j9"
server_key: "ace1c4d91dc14cb8944e1ab8887fd2fd"
server_key_iv: "e5bc5afb9672fecd48d9b5218483627f"
server_key_version: "encryptionkey@0"
I20260924 16:12:52.900144 4242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:52.923270 4242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:52.927295 4242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:52.927587 4242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:52.928560 4242 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:12:52.929935 4242 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:12:52.930040 4242 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:52.930189 4242 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:12:52.930259 4242 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:52.964389 4242 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.3:42433
I20260924 16:12:52.964442 4371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.3:42433 every 8 connection(s)
I20260924 16:12:52.965777 4242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
I20260924 16:12:52.966946 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 4242
I20260924 16:12:52.967131 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/minicluster-data/ts-2/wal/instance
I20260924 16:12:52.967675 30632 external_mini_cluster.cc:1428] Setting key 86cbeef337eb6692be643092a255f8d7
I20260924 16:12:52.978811 4100 heartbeater.cc:503] Master 127.29.234.62:35867 was elected leader, sending a full tablet report...
Sep 24 16:12:52 dist-test-slave-v5j9 krb5kdc[3862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266372, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.3@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:12:52.987087 4102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:52.968891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.3:39057 (local address 127.29.234.62:35867)
0924 16:12:52.969106 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:52.969112 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:52.970574 (+ 1462us) server_negotiation.cc:407] Connection header received
0924 16:12:52.972443 (+ 1869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:52.972449 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:52.972546 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:52.972628 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:52.974196 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:52.974986 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:52.976528 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:52.976936 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:52.977041 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:52.981016 (+ 3975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:52.981021 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:52.981025 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:52.981053 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:52.984264 (+ 3211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:52.984952 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:52.984956 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:52.984958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:52.985083 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:52.985604 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:52.985608 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:52.985611 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:52.985889 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:52.986012 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:52.986681 (+ 669us) server_negotiation.cc:299] Negotiation successful
0924 16:12:52.986840 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 16:12:52.988087 4374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:52.969474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:35867 (local address 127.29.234.3:39057)
0924 16:12:52.970401 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:52.970449 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:52.972162 (+ 1713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:52.972846 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:52.972856 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:52.973334 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:52.974032 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:52.974052 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:52.975116 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:52.975120 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:52.976379 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:52.976391 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:52.977166 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:52.977171 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:52.977464 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:52.978178 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:52.978242 (+ 64us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:52.980843 (+ 2601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:52.984389 (+ 3546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:52.984396 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:52.984407 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:52.984838 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:52.985218 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:52.985223 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:52.985226 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:52.985464 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:52.986002 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:52.986008 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:52.986328 (+ 320us) client_negotiation.cc:769] Sending connection context
0924 16:12:52.986590 (+ 262us) client_negotiation.cc:240] Negotiation successful
0924 16:12:52.986978 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":360,"threads_started":1}
I20260924 16:12:52.989538 4372 heartbeater.cc:347] Connected to a master server at 127.29.234.62:35867
I20260924 16:12:52.989810 4372 heartbeater.cc:464] Registering TS with master...
I20260924 16:12:52.990459 4372 heartbeater.cc:511] Master 127.29.234.62:35867 requested a full tablet report, sending...
I20260924 16:12:52.991765 3907 ts_manager.cc:194] Registered new tserver with Master: 0851b8f78bc74f27b871f6530ac6ae3e (127.29.234.3:42433)
I20260924 16:12:52.992568 3907 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.234.3@KRBTEST.COM'} at 127.29.234.3:39057
I20260924 16:12:52.995721 30632 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/krb5kdc/test-admin.keytab.
Sep 24 16:12:53 dist-test-slave-v5j9 krb5kdc[3862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266373, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:53.022930 30632 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260924 16:12:53.028283 4379 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:12:53.025701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60322 (local address 127.29.234.1:32953)
0924 16:12:53.026441 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:53.026449 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:53.026475 (+ 26us) server_negotiation.cc:407] Connection header received
0924 16:12:53.026537 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:53.026544 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:53.026771 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:53.026945 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:53.027811 (+ 866us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60322: BlockingRecv error: recv got EOF from 127.0.0.1:60322 (error 108)
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":368,"threads_started":1}
W20260924 16:12:53.028330 30632 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/krb5kdc/krb5cc' of type FILE
Sep 24 16:12:53 dist-test-slave-v5j9 krb5kdc[3862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266373, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:53.030018 30632 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 16:12:53 dist-test-slave-v5j9 krb5kdc[3862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266373, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:53.031358 30632 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 16:12:53 dist-test-slave-v5j9 krb5kdc[3862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266373, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.234.1@KRBTEST.COM
I20260924 16:12:53.047104 4379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:53.033014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60330 (local address 127.29.234.1:32953)
0924 16:12:53.033227 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:53.033232 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:53.033395 (+ 163us) server_negotiation.cc:407] Connection header received
0924 16:12:53.033622 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:53.033626 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:53.033685 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:53.033790 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:53.034792 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:53.035765 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:53.037046 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:53.037684 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:53.037829 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:53.041334 (+ 3505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:53.041386 (+ 52us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:53.041410 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:53.041460 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:53.044115 (+ 2655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:53.044868 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:53.044884 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:53.044886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:53.045012 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:53.045453 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:53.045462 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:53.045464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:53.045825 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:53.046173 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:53.046484 (+ 311us) server_negotiation.cc:299] Negotiation successful
0924 16:12:53.046783 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260924 16:12:53.052981 4379 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:12:53.051162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60342 (local address 127.29.234.1:32953)
0924 16:12:53.051392 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:53.051397 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:53.051585 (+ 188us) server_negotiation.cc:407] Connection header received
0924 16:12:53.051706 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:53.051709 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:53.051764 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:53.051839 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:53.052681 (+ 842us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60342: BlockingRecv error: recv got EOF from 127.0.0.1:60342 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
W20260924 16:12:53.053051 30632 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/krb5kdc/krb5cc' of type FILE
Sep 24 16:12:53 dist-test-slave-v5j9 krb5kdc[3862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266373, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:53.054813 30632 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 16:12:53 dist-test-slave-v5j9 krb5kdc[3862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266373, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:53.056393 30632 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 16:12:53 dist-test-slave-v5j9 krb5kdc[3862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266373, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.234.1@KRBTEST.COM
I20260924 16:12:53.071974 4379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:53.058024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60344 (local address 127.29.234.1:32953)
0924 16:12:53.058316 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:53.058323 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:53.058576 (+ 253us) server_negotiation.cc:407] Connection header received
0924 16:12:53.058819 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:53.058823 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:53.058887 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:53.058995 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:53.059984 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:53.060777 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:53.062153 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:53.062601 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:53.062761 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:53.066612 (+ 3851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:53.066623 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:53.066627 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:53.066666 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:53.069423 (+ 2757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:53.070261 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:53.070265 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:53.070267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:53.070393 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:53.070828 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:53.070832 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:53.070834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:53.071109 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:53.071315 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:53.071575 (+ 260us) server_negotiation.cc:299] Negotiation successful
0924 16:12:53.071738 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
W20260924 16:12:53.076768 4379 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:12:53.074882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60358 (local address 127.29.234.1:32953)
0924 16:12:53.075192 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:53.075198 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:53.075402 (+ 204us) server_negotiation.cc:407] Connection header received
0924 16:12:53.075608 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:53.075614 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:53.075705 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:53.075807 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:53.076511 (+ 704us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60358: BlockingRecv error: recv got EOF from 127.0.0.1:60358 (error 108)
Metrics: {"server-negotiator.queue_time_us":170}
W20260924 16:12:53.076917 30632 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790266323169505-30632-0/krb5kdc/krb5cc' of type FILE
Sep 24 16:12:53 dist-test-slave-v5j9 krb5kdc[3862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266373, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:53.078521 30632 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 16:12:53 dist-test-slave-v5j9 krb5kdc[3862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266373, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:53.079722 30632 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 16:12:53 dist-test-slave-v5j9 krb5kdc[3862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266373, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.234.1@KRBTEST.COM
I20260924 16:12:53.093766 4379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:53.080934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60374 (local address 127.29.234.1:32953)
0924 16:12:53.081145 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:53.081153 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:53.081504 (+ 351us) server_negotiation.cc:407] Connection header received
0924 16:12:53.081686 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:53.081691 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:53.081778 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:53.081875 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:53.082791 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:53.083480 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:53.084682 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:53.085102 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:53.085233 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:53.088721 (+ 3488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:53.088727 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:53.088732 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:53.088773 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:53.091222 (+ 2449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:53.091853 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:53.091859 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:53.091862 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:53.092013 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:53.092374 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:53.092378 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:53.092382 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:53.092631 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:53.092960 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:53.093218 (+ 258us) server_negotiation.cc:299] Negotiation successful
0924 16:12:53.093456 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 16:12:53.095376 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 3969
I20260924 16:12:53.107666 4379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:53.096041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60390 (local address 127.29.234.1:32953)
0924 16:12:53.096215 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:53.096223 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:53.096532 (+ 309us) server_negotiation.cc:407] Connection header received
0924 16:12:53.096715 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:53.096720 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:53.096785 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:53.096881 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:53.097821 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:53.098477 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:53.099661 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:53.100102 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:53.100236 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:53.102255 (+ 2019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:53.102261 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:53.102265 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:53.102305 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:53.104996 (+ 2691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:53.105752 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:53.105757 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:53.105761 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:53.105905 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:53.106286 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:53.106290 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:53.106294 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:53.106552 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:53.106833 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:53.107111 (+ 278us) server_negotiation.cc:299] Negotiation successful
0924 16:12:53.107326 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 16:12:53.108274 4094 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:12:53.190443 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 3969
I20260924 16:12:53.208029 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 4106
Sep 24 16:12:53 dist-test-slave-v5j9 krb5kdc[3862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266373, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.234.2@KRBTEST.COM
I20260924 16:12:53.224174 4392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:53.208928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47318 (local address 127.29.234.2:46711)
0924 16:12:53.209544 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:53.209551 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:53.209578 (+ 27us) server_negotiation.cc:407] Connection header received
0924 16:12:53.209650 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:53.209657 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:53.209867 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:53.209997 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:53.210919 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:53.211907 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:53.213151 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:53.213900 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:53.214020 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:53.218063 (+ 4043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:53.218076 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:53.218094 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:53.218131 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:53.221132 (+ 3001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:53.222015 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:53.222023 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:53.222026 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:53.222149 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:53.222560 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:53.222565 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:53.222571 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:53.222926 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:53.223255 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:53.223596 (+ 341us) server_negotiation.cc:299] Negotiation successful
0924 16:12:53.223862 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":313,"threads_started":1}
I20260924 16:12:53.225543 4232 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:12:53.310086 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 4106
I20260924 16:12:53.327080 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 4242
Sep 24 16:12:53 dist-test-slave-v5j9 krb5kdc[3862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266373, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.234.3@KRBTEST.COM
I20260924 16:12:53.343179 4394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:53.328283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47462 (local address 127.29.234.3:42433)
0924 16:12:53.328903 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:53.328910 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:53.328939 (+ 29us) server_negotiation.cc:407] Connection header received
0924 16:12:53.329002 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:53.329018 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:53.329242 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:53.329444 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:53.330544 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:53.331573 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:53.332862 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:53.333627 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:53.333761 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:53.337807 (+ 4046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:53.337817 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:53.337833 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:53.337868 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:53.340458 (+ 2590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:53.341123 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:53.341131 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:53.341133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:53.341252 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:53.341644 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:53.341651 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:53.341654 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:53.341976 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:53.342304 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:53.342583 (+ 279us) server_negotiation.cc:299] Negotiation successful
0924 16:12:53.342905 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":332,"threads_started":1}
I20260924 16:12:53.344486 4367 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:12:53.427587 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 4242
I20260924 16:12:53.444602 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 3875
I20260924 16:12:53.445964 3938 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:12:53.552211 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 3875
2026-09-24T16:12:53Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4011 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:12:53 dist-test-slave-v5j9 krb5kdc[4399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:12:53 dist-test-slave-v5j9 krb5kdc[4399](info): set up 2 sockets
Sep 24 16:12:53 dist-test-slave-v5j9 krb5kdc[4399](info): commencing operation
krb5kdc: starting...
W20260924 16:12:54.657295 30632 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Sep 24 16:12:54 dist-test-slave-v5j9 krb5kdc[4399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266374, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:12:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:12:54Z Disabled control of system clock
WARNING: no policy specified for oryx/127.29.234.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.234.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.234.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/krb5kdc/oryx_127.29.234.62.keytab.
Entry for principal oryx/127.29.234.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/krb5kdc/oryx_127.29.234.62.keytab.
I20260924 16:12:54.872290 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:40099
--webserver_interface=127.29.234.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/krb5kdc/oryx_127.29.234.62.keytab
--principal=oryx/127.29.234.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:40243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.234.62:40099
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:12:55.199570 4412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:55.199859 4412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:55.200037 4412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:55.207095 4412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:12:55.207194 4412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:55.207319 4412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:55.207419 4412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:12:55.207499 4412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:12:55.217465 4412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:40243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.234.62:40099
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/krb5kdc/oryx_127.29.234.62.keytab
--principal=oryx/127.29.234.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:40099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.234.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:12:55.218873 4412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:55.220212 4412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:55.232506 4423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:55.232580 4421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:55.232602 4420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:55.234957 4412 server_base.cc:1034] running on GCE node
I20260924 16:12:55.235816 4412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:55.237262 4412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:55.238536 4412 hybrid_clock.cc:648] HybridClock initialized: now 1790266375238478 us; error 52 us; skew 500 ppm
I20260924 16:12:55.238914 4412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:12:55 dist-test-slave-v5j9 krb5kdc[4399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266375, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.234.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:55.242163 4412 init.cc:377] Logged in from keytab as oryx/127.29.234.62@KRBTEST.COM (short username oryx)
I20260924 16:12:55.243773 4412 webserver.cc:460] Webserver started at http://127.29.234.62:43109/ using document root <none> and password file <none>
I20260924 16:12:55.244531 4412 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:55.244685 4412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:55.245005 4412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:55.248221 4412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/master-0/data/instance:
uuid: "d3639f056333442f859ccc6c600711ac"
format_stamp: "Formatted at 2026-09-24 16:12:55 on dist-test-slave-v5j9"
server_key: "c0bfefb23710ab9f8529d62cc7d34243"
server_key_iv: "938e5ddc44b2a6f3a9a588495e6599cb"
server_key_version: "encryptionkey@0"
I20260924 16:12:55.249042 4412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/master-0/wal/instance:
uuid: "d3639f056333442f859ccc6c600711ac"
format_stamp: "Formatted at 2026-09-24 16:12:55 on dist-test-slave-v5j9"
server_key: "c0bfefb23710ab9f8529d62cc7d34243"
server_key_iv: "938e5ddc44b2a6f3a9a588495e6599cb"
server_key_version: "encryptionkey@0"
I20260924 16:12:55.254732 4412 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 16:12:55.258354 4430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:55.259738 4412 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:12:55.259931 4412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/master-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/master-0/wal
uuid: "d3639f056333442f859ccc6c600711ac"
format_stamp: "Formatted at 2026-09-24 16:12:55 on dist-test-slave-v5j9"
server_key: "c0bfefb23710ab9f8529d62cc7d34243"
server_key_iv: "938e5ddc44b2a6f3a9a588495e6599cb"
server_key_version: "encryptionkey@0"
I20260924 16:12:55.260138 4412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:55.288198 4412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:55.292932 4412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:55.293299 4412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:55.312055 4412 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.62:40099
I20260924 16:12:55.312074 4481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.62:40099 every 8 connection(s)
I20260924 16:12:55.313699 4412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
I20260924 16:12:55.314129 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 4412
I20260924 16:12:55.314369 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/master-0/wal/instance
I20260924 16:12:55.315299 30632 external_mini_cluster.cc:1428] Setting key ea95c5981d3a81b5af03fc06edf96869
I20260924 16:12:55.318554 4482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 16:12:55 dist-test-slave-v5j9 krb5kdc[4399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266374, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM
I20260924 16:12:55.330895 4482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3639f056333442f859ccc6c600711ac: Bootstrap starting.
I20260924 16:12:55.334997 4482 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3639f056333442f859ccc6c600711ac: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:55.336302 4482 log.cc:824] T 00000000000000000000000000000000 P d3639f056333442f859ccc6c600711ac: Log is configured to *not* fsync() on all Append() calls
I20260924 16:12:55.339740 4482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3639f056333442f859ccc6c600711ac: No bootstrap required, opened a new log
I20260924 16:12:55.339953 4485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:55.317431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36972 (local address 127.29.234.62:40099)
0924 16:12:55.318266 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:55.318282 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:55.318332 (+ 50us) server_negotiation.cc:407] Connection header received
0924 16:12:55.320622 (+ 2290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:55.320651 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:55.321123 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:55.321574 (+ 451us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:55.322947 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:55.325075 (+ 2128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:55.326429 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:55.327428 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:55.327626 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:55.331428 (+ 3802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:55.331438 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:55.331455 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:55.331498 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:55.335536 (+ 4038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:55.336224 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:55.336234 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:55.336245 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:55.336447 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:55.336886 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:55.336898 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:55.336904 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:55.337547 (+ 643us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:55.337867 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:55.338218 (+ 351us) server_negotiation.cc:299] Negotiation successful
0924 16:12:55.338641 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":317,"threads_started":1}
I20260924 16:12:55.347760 4482 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3639f056333442f859ccc6c600711ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3639f056333442f859ccc6c600711ac" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 40099 } }
I20260924 16:12:55.348068 4482 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3639f056333442f859ccc6c600711ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:55.348138 4482 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3639f056333442f859ccc6c600711ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3639f056333442f859ccc6c600711ac, State: Initialized, Role: FOLLOWER
I20260924 16:12:55.348591 4482 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3639f056333442f859ccc6c600711ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3639f056333442f859ccc6c600711ac" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 40099 } }
I20260924 16:12:55.348783 4482 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3639f056333442f859ccc6c600711ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:12:55.348868 4482 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3639f056333442f859ccc6c600711ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:12:55.349000 4482 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3639f056333442f859ccc6c600711ac [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:55.351328 4482 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3639f056333442f859ccc6c600711ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3639f056333442f859ccc6c600711ac" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 40099 } }
I20260924 16:12:55.351751 4482 leader_election.cc:302] T 00000000000000000000000000000000 P d3639f056333442f859ccc6c600711ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3639f056333442f859ccc6c600711ac; no voters:
I20260924 16:12:55.352360 4482 leader_election.cc:288] T 00000000000000000000000000000000 P d3639f056333442f859ccc6c600711ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:12:55.352490 4487 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3639f056333442f859ccc6c600711ac [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:12:55.353188 4487 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3639f056333442f859ccc6c600711ac [term 1 LEADER]: Becoming Leader. State: Replica: d3639f056333442f859ccc6c600711ac, State: Running, Role: LEADER
I20260924 16:12:55.353799 4487 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3639f056333442f859ccc6c600711ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3639f056333442f859ccc6c600711ac" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 40099 } }
I20260924 16:12:55.354599 4482 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3639f056333442f859ccc6c600711ac [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:12:55.356717 4489 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3639f056333442f859ccc6c600711ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3639f056333442f859ccc6c600711ac. Latest consensus state: current_term: 1 leader_uuid: "d3639f056333442f859ccc6c600711ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3639f056333442f859ccc6c600711ac" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 40099 } } }
I20260924 16:12:55.356688 4488 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3639f056333442f859ccc6c600711ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3639f056333442f859ccc6c600711ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3639f056333442f859ccc6c600711ac" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 40099 } } }
I20260924 16:12:55.357077 4489 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3639f056333442f859ccc6c600711ac [sys.catalog]: This master's current role is: LEADER
I20260924 16:12:55.357182 4488 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3639f056333442f859ccc6c600711ac [sys.catalog]: This master's current role is: LEADER
I20260924 16:12:55.357915 4493 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:12:55.363543 4493 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:12:55.372804 4493 catalog_manager.cc:1347] Generated new cluster ID: e95643b1533745a68073f27a96aa4b1a
I20260924 16:12:55.372920 4493 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:12:55.394760 4493 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:12:55.396701 4493 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:12:55.406955 4493 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3639f056333442f859ccc6c600711ac: Generated new TSK 0
I20260924 16:12:55.408085 4493 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.29.234.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.234.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.234.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/krb5kdc/oryx_127.29.234.1.keytab.
Entry for principal oryx/127.29.234.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/krb5kdc/oryx_127.29.234.1.keytab.
I20260924 16:12:55.468514 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.1:0
--local_ip_for_outbound_sockets=127.29.234.1
--webserver_interface=127.29.234.1
--webserver_port=0
--tserver_master_addrs=127.29.234.62:40099
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/krb5kdc/oryx_127.29.234.1.keytab
--principal=oryx/127.29.234.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:40243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:12:55.798683 4508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:55.798959 4508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:55.799130 4508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:55.806277 4508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:55.806420 4508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:55.806609 4508 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.1
I20260924 16:12:55.816104 4508 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:40243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/krb5kdc/oryx_127.29.234.1.keytab
--principal=oryx/127.29.234.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.234.1
--webserver_port=0
--tserver_master_addrs=127.29.234.62:40099
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.1
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:12:55.817518 4508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:55.818908 4508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:55.832469 4513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:55.832415 4514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:55.832499 4516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:55.833599 4508 server_base.cc:1034] running on GCE node
I20260924 16:12:55.834177 4508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:55.835038 4508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:55.836236 4508 hybrid_clock.cc:648] HybridClock initialized: now 1790266375836202 us; error 34 us; skew 500 ppm
I20260924 16:12:55.836656 4508 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:12:55 dist-test-slave-v5j9 krb5kdc[4399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266375, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.234.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:55.841197 4508 init.cc:377] Logged in from keytab as oryx/127.29.234.1@KRBTEST.COM (short username oryx)
I20260924 16:12:55.843736 4508 webserver.cc:460] Webserver started at http://127.29.234.1:32803/ using document root <none> and password file <none>
I20260924 16:12:55.844753 4508 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:55.844913 4508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:55.845300 4508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:55.848724 4508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-0/data/instance:
uuid: "76eabf034e914b95887d119fd3f8faa3"
format_stamp: "Formatted at 2026-09-24 16:12:55 on dist-test-slave-v5j9"
server_key: "e5f2615cff6d714a277cc84373f77b09"
server_key_iv: "11270d22460670904b1f8dd2c5adb700"
server_key_version: "encryptionkey@0"
I20260924 16:12:55.849566 4508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-0/wal/instance:
uuid: "76eabf034e914b95887d119fd3f8faa3"
format_stamp: "Formatted at 2026-09-24 16:12:55 on dist-test-slave-v5j9"
server_key: "e5f2615cff6d714a277cc84373f77b09"
server_key_iv: "11270d22460670904b1f8dd2c5adb700"
server_key_version: "encryptionkey@0"
I20260924 16:12:55.854738 4508 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 16:12:55.858286 4523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:55.859671 4508 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:12:55.859845 4508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-0/wal
uuid: "76eabf034e914b95887d119fd3f8faa3"
format_stamp: "Formatted at 2026-09-24 16:12:55 on dist-test-slave-v5j9"
server_key: "e5f2615cff6d714a277cc84373f77b09"
server_key_iv: "11270d22460670904b1f8dd2c5adb700"
server_key_version: "encryptionkey@0"
I20260924 16:12:55.860051 4508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:55.874312 4508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:55.878237 4508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:55.878552 4508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:55.879515 4508 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:12:55.880959 4508 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:12:55.881053 4508 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:55.881181 4508 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:12:55.881268 4508 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:55.914572 4508 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.1:40063
I20260924 16:12:55.914608 4636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.1:40063 every 8 connection(s)
I20260924 16:12:55.915875 4508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
I20260924 16:12:55.919039 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 4508
I20260924 16:12:55.919283 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-0/wal/instance
I20260924 16:12:55.920150 30632 external_mini_cluster.cc:1428] Setting key cfd84b76d5475b600d56e26959dd5123
Sep 24 16:12:55 dist-test-slave-v5j9 krb5kdc[4399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266375, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.234.1@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM
I20260924 16:12:55.936187 4639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:55.918497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.1:52107 (local address 127.29.234.62:40099)
0924 16:12:55.918934 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:55.918938 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:55.920101 (+ 1163us) server_negotiation.cc:407] Connection header received
0924 16:12:55.922216 (+ 2115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:55.922221 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:55.922303 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:55.922426 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:55.923932 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:55.924634 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:55.926309 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:55.926706 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:55.926837 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:55.930408 (+ 3571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:55.930413 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:55.930417 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:55.930453 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:55.932956 (+ 2503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:55.933795 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:55.933801 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:55.933805 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:55.933935 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:55.934368 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:55.934373 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:55.934377 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:55.934640 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:55.934798 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:55.935564 (+ 766us) server_negotiation.cc:299] Negotiation successful
0924 16:12:55.935816 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":240,"threads_started":1}
I20260924 16:12:55.937261 4640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:55.919182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:40099 (local address 127.29.234.1:52107)
0924 16:12:55.919947 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:55.919991 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:55.921924 (+ 1933us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:55.922665 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:55.922673 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:55.923175 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:55.923757 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:55.923774 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:55.924793 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:55.924806 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:55.926154 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:55.926166 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:55.926942 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:55.926949 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:55.927185 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:55.927909 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:55.927933 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:55.930149 (+ 2216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:55.933110 (+ 2961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:55.933120 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:55.933140 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:55.933653 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:55.934046 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:55.934051 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:55.934055 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:55.934247 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:55.934778 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:55.934786 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:55.935145 (+ 359us) client_negotiation.cc:769] Sending connection context
0924 16:12:55.935437 (+ 292us) client_negotiation.cc:240] Negotiation successful
0924 16:12:55.936060 (+ 623us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":303,"threads_started":1}
I20260924 16:12:55.938967 4637 heartbeater.cc:347] Connected to a master server at 127.29.234.62:40099
I20260924 16:12:55.939333 4637 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for oryx/127.29.234.2@KRBTEST.COM; defaulting to no policy
I20260924 16:12:55.940335 4637 heartbeater.cc:511] Master 127.29.234.62:40099 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.234.2@KRBTEST.COM" created.
I20260924 16:12:55.942636 4447 ts_manager.cc:194] Registered new tserver with Master: 76eabf034e914b95887d119fd3f8faa3 (127.29.234.1:40063)
I20260924 16:12:55.944108 4447 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.234.1@KRBTEST.COM'} at 127.29.234.1:52107
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.234.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/krb5kdc/oryx_127.29.234.2.keytab.
Entry for principal oryx/127.29.234.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/krb5kdc/oryx_127.29.234.2.keytab.
I20260924 16:12:55.968778 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.2:0
--local_ip_for_outbound_sockets=127.29.234.2
--webserver_interface=127.29.234.2
--webserver_port=0
--tserver_master_addrs=127.29.234.62:40099
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/krb5kdc/oryx_127.29.234.2.keytab
--principal=oryx/127.29.234.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:40243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:12:56.295567 4643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:56.295866 4643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:56.296026 4643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:56.303153 4643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:56.303305 4643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:56.303527 4643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.2
I20260924 16:12:56.313409 4643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:40243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/krb5kdc/oryx_127.29.234.2.keytab
--principal=oryx/127.29.234.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.234.2
--webserver_port=0
--tserver_master_addrs=127.29.234.62:40099
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.2
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:12:56.314729 4643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:56.316037 4643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:56.330408 4648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:56.330463 4649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:56.330482 4651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:56.331661 4643 server_base.cc:1034] running on GCE node
I20260924 16:12:56.332388 4643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:56.333456 4643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:56.334703 4643 hybrid_clock.cc:648] HybridClock initialized: now 1790266376334644 us; error 52 us; skew 500 ppm
I20260924 16:12:56.335047 4643 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:12:56 dist-test-slave-v5j9 krb5kdc[4399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266376, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.234.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:56.338284 4643 init.cc:377] Logged in from keytab as oryx/127.29.234.2@KRBTEST.COM (short username oryx)
I20260924 16:12:56.339746 4643 webserver.cc:460] Webserver started at http://127.29.234.2:42535/ using document root <none> and password file <none>
I20260924 16:12:56.340483 4643 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:56.340632 4643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:56.340945 4643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:56.343930 4643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-1/data/instance:
uuid: "14d33685ba3849c78f336923bf49e2d4"
format_stamp: "Formatted at 2026-09-24 16:12:56 on dist-test-slave-v5j9"
server_key: "07f9e816ca47ba80f628e3bceaababd9"
server_key_iv: "d92eadb560ea8760d9a5d5fe8ee84c64"
server_key_version: "encryptionkey@0"
I20260924 16:12:56.344731 4643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-1/wal/instance:
uuid: "14d33685ba3849c78f336923bf49e2d4"
format_stamp: "Formatted at 2026-09-24 16:12:56 on dist-test-slave-v5j9"
server_key: "07f9e816ca47ba80f628e3bceaababd9"
server_key_iv: "d92eadb560ea8760d9a5d5fe8ee84c64"
server_key_version: "encryptionkey@0"
I20260924 16:12:56.350034 4643 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 16:12:56.353595 4658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:56.355100 4643 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:12:56.355279 4643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-1/wal
uuid: "14d33685ba3849c78f336923bf49e2d4"
format_stamp: "Formatted at 2026-09-24 16:12:56 on dist-test-slave-v5j9"
server_key: "07f9e816ca47ba80f628e3bceaababd9"
server_key_iv: "d92eadb560ea8760d9a5d5fe8ee84c64"
server_key_version: "encryptionkey@0"
I20260924 16:12:56.355480 4643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:56.373665 4643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:56.377473 4643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:56.377781 4643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:56.378690 4643 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:12:56.380038 4643 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:12:56.380141 4643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:56.380301 4643 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:12:56.380375 4643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:56.413693 4643 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.2:39763
I20260924 16:12:56.413735 4774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.2:39763 every 8 connection(s)
I20260924 16:12:56.414966 4643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
I20260924 16:12:56.418165 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 4643
I20260924 16:12:56.418330 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-1/wal/instance
I20260924 16:12:56.418785 30632 external_mini_cluster.cc:1428] Setting key 2dd3c23ce06d90aadc02c996c08181f3
Sep 24 16:12:56 dist-test-slave-v5j9 krb5kdc[4399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266376, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.234.2@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM
I20260924 16:12:56.435626 4639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:56.417524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.2:60271 (local address 127.29.234.62:40099)
0924 16:12:56.417692 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:56.417697 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:56.419061 (+ 1364us) server_negotiation.cc:407] Connection header received
0924 16:12:56.421117 (+ 2056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:56.421122 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:56.421205 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:56.421329 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:56.422910 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:56.424053 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:56.425680 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:56.426456 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:56.426590 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:56.430031 (+ 3441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:56.430036 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:56.430039 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:56.430081 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:56.432623 (+ 2542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:56.433327 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:56.433332 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:56.433337 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:56.433510 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:56.433965 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:56.433970 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:56.433973 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:56.434232 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:56.434393 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:56.435041 (+ 648us) server_negotiation.cc:299] Negotiation successful
0924 16:12:56.435257 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 16:12:56.436592 4777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:56.418035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:40099 (local address 127.29.234.2:60271)
0924 16:12:56.418887 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:56.418931 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:56.420860 (+ 1929us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:56.421598 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:56.421606 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:56.422127 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:56.422725 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:56.422742 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:56.424216 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:56.424220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:56.425545 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:56.425557 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:56.426668 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:56.426673 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:56.426896 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:56.427602 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:56.427628 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:56.429797 (+ 2169us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:56.432779 (+ 2982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:56.432788 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:56.432803 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:56.433202 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:56.433623 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:56.433628 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:56.433632 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:56.433812 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:56.434358 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:56.434366 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:56.434691 (+ 325us) client_negotiation.cc:769] Sending connection context
0924 16:12:56.434954 (+ 263us) client_negotiation.cc:240] Negotiation successful
0924 16:12:56.435476 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":367,"threads_started":1}
I20260924 16:12:56.438335 4775 heartbeater.cc:347] Connected to a master server at 127.29.234.62:40099
I20260924 16:12:56.438666 4775 heartbeater.cc:464] Registering TS with master...
I20260924 16:12:56.439394 4775 heartbeater.cc:511] Master 127.29.234.62:40099 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.29.234.3@KRBTEST.COM; defaulting to no policy
I20260924 16:12:56.440907 4447 ts_manager.cc:194] Registered new tserver with Master: 14d33685ba3849c78f336923bf49e2d4 (127.29.234.2:39763)
I20260924 16:12:56.441883 4447 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.234.2@KRBTEST.COM'} at 127.29.234.2:60271
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.234.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.234.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/krb5kdc/oryx_127.29.234.3.keytab.
Entry for principal oryx/127.29.234.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/krb5kdc/oryx_127.29.234.3.keytab.
I20260924 16:12:56.469210 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.3:0
--local_ip_for_outbound_sockets=127.29.234.3
--webserver_interface=127.29.234.3
--webserver_port=0
--tserver_master_addrs=127.29.234.62:40099
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/krb5kdc/oryx_127.29.234.3.keytab
--principal=oryx/127.29.234.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:40243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:12:56.795122 4780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:56.795423 4780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:56.795522 4780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:56.802778 4780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:56.802884 4780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:56.802994 4780 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.3
I20260924 16:12:56.812618 4780 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:40243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/krb5kdc/oryx_127.29.234.3.keytab
--principal=oryx/127.29.234.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.234.3
--webserver_port=0
--tserver_master_addrs=127.29.234.62:40099
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.3
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:12:56.813956 4780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:56.815239 4780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:56.828802 4785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:56.828872 4788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:56.828835 4786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:56.830053 4780 server_base.cc:1034] running on GCE node
I20260924 16:12:56.830706 4780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:56.831683 4780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:56.832922 4780 hybrid_clock.cc:648] HybridClock initialized: now 1790266376832870 us; error 43 us; skew 500 ppm
I20260924 16:12:56.833257 4780 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:12:56 dist-test-slave-v5j9 krb5kdc[4399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266376, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.234.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:56.836629 4780 init.cc:377] Logged in from keytab as oryx/127.29.234.3@KRBTEST.COM (short username oryx)
I20260924 16:12:56.838236 4780 webserver.cc:460] Webserver started at http://127.29.234.3:42061/ using document root <none> and password file <none>
I20260924 16:12:56.838948 4780 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:56.839093 4780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:56.839408 4780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:56.842420 4780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-2/data/instance:
uuid: "20193887ad994ccb8f3e9a4e4cad037c"
format_stamp: "Formatted at 2026-09-24 16:12:56 on dist-test-slave-v5j9"
server_key: "f5cd7d7be3d52eafc0569b6d4d35503e"
server_key_iv: "059ff158896fd8cc9e700f9be33b84f6"
server_key_version: "encryptionkey@0"
I20260924 16:12:56.843223 4780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-2/wal/instance:
uuid: "20193887ad994ccb8f3e9a4e4cad037c"
format_stamp: "Formatted at 2026-09-24 16:12:56 on dist-test-slave-v5j9"
server_key: "f5cd7d7be3d52eafc0569b6d4d35503e"
server_key_iv: "059ff158896fd8cc9e700f9be33b84f6"
server_key_version: "encryptionkey@0"
I20260924 16:12:56.848748 4780 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 16:12:56.852289 4795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:56.853983 4780 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:12:56.854169 4780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-2/wal
uuid: "20193887ad994ccb8f3e9a4e4cad037c"
format_stamp: "Formatted at 2026-09-24 16:12:56 on dist-test-slave-v5j9"
server_key: "f5cd7d7be3d52eafc0569b6d4d35503e"
server_key_iv: "059ff158896fd8cc9e700f9be33b84f6"
server_key_version: "encryptionkey@0"
I20260924 16:12:56.854398 4780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:56.887804 4780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:56.892377 4780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:56.892689 4780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:56.893782 4780 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:12:56.895228 4780 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:12:56.895337 4780 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:56.895489 4780 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:12:56.895562 4780 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:56.928418 4780 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.3:34375
I20260924 16:12:56.928514 4908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.3:34375 every 8 connection(s)
I20260924 16:12:56.929804 4780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
I20260924 16:12:56.930478 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 4780
I20260924 16:12:56.930670 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-2/wal/instance
I20260924 16:12:56.931257 30632 external_mini_cluster.cc:1428] Setting key dfe75751c9ff0485ea7cb147671f7a14
Sep 24 16:12:56 dist-test-slave-v5j9 krb5kdc[4399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266376, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.234.3@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM
I20260924 16:12:56.946969 4637 heartbeater.cc:503] Master 127.29.234.62:40099 was elected leader, sending a full tablet report...
I20260924 16:12:56.949769 4639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:56.932803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.3:48663 (local address 127.29.234.62:40099)
0924 16:12:56.932974 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:56.932979 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:56.934248 (+ 1269us) server_negotiation.cc:407] Connection header received
0924 16:12:56.936100 (+ 1852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:56.936104 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:56.936195 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:56.936282 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:56.937878 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:56.938626 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:56.940070 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:56.940439 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:56.940527 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:56.943731 (+ 3204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:56.943736 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:56.943739 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:56.943769 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:56.946608 (+ 2839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:56.947500 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:56.947505 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:56.947508 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:56.947674 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:56.948151 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:56.948154 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:56.948156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:56.948523 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:56.948665 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:56.949280 (+ 615us) server_negotiation.cc:299] Negotiation successful
0924 16:12:56.949508 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 16:12:56.950747 4911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:56.933211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:40099 (local address 127.29.234.3:48663)
0924 16:12:56.934103 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:56.934148 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:56.935843 (+ 1695us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:56.936486 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:56.936493 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:56.937019 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:56.937740 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:56.937758 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:56.938745 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:56.938749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:56.939948 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:56.939964 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:56.940617 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:56.940621 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:56.940815 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:56.941491 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:56.941512 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:56.943548 (+ 2036us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:56.946782 (+ 3234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:56.946789 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:56.946800 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:56.947367 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:56.947796 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:56.947801 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:56.947804 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:56.948040 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:56.948651 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:56.948657 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:56.948947 (+ 290us) client_negotiation.cc:769] Sending connection context
0924 16:12:56.949161 (+ 214us) client_negotiation.cc:240] Negotiation successful
0924 16:12:56.949615 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":337,"threads_started":1}
I20260924 16:12:56.952113 4909 heartbeater.cc:347] Connected to a master server at 127.29.234.62:40099
I20260924 16:12:56.952435 4909 heartbeater.cc:464] Registering TS with master...
I20260924 16:12:56.953161 4909 heartbeater.cc:511] Master 127.29.234.62:40099 requested a full tablet report, sending...
I20260924 16:12:56.954586 4447 ts_manager.cc:194] Registered new tserver with Master: 20193887ad994ccb8f3e9a4e4cad037c (127.29.234.3:34375)
I20260924 16:12:56.955490 4447 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.234.3@KRBTEST.COM'} at 127.29.234.3:48663
I20260924 16:12:56.959227 30632 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.29.234.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.234.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.234.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/krb5kdc/oryx_127.29.234.61.keytab.
Entry for principal oryx/127.29.234.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/krb5kdc/oryx_127.29.234.61.keytab.
I20260924 16:12:57.009028 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.61:42395
--webserver_interface=127.29.234.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/krb5kdc/oryx_127.29.234.61.keytab
--principal=oryx/127.29.234.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:40243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.234.62:40099,127.29.234.61:42395
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:12:57.340588 4914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:57.340948 4914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:57.341100 4914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:57.348443 4914 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:12:57.348613 4914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:57.348708 4914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:57.348801 4914 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:12:57.348891 4914 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:12:57.359368 4914 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:40243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.29.234.62:40099,127.29.234.61:42395
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/krb5kdc/oryx_127.29.234.61.keytab
--principal=oryx/127.29.234.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.61:42395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.29.234.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
W20260924 16:12:57.359786 4914 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.29.234.62:40099,127.29.234.61:42395'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260924 16:12:57.360947 4914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:57.362416 4914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:57.374785 4919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:57.374785 4920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:57.374792 4922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:57.375242 4914 server_base.cc:1034] running on GCE node
I20260924 16:12:57.376246 4914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:57.377710 4914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:57.378948 4914 hybrid_clock.cc:648] HybridClock initialized: now 1790266377378873 us; error 58 us; skew 500 ppm
I20260924 16:12:57.379282 4914 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:12:57 dist-test-slave-v5j9 krb5kdc[4399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266377, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.234.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:57.382468 4914 init.cc:377] Logged in from keytab as oryx/127.29.234.61@KRBTEST.COM (short username oryx)
I20260924 16:12:57.383868 4914 webserver.cc:460] Webserver started at http://127.29.234.61:39055/ using document root <none> and password file <none>
I20260924 16:12:57.384577 4914 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:57.384706 4914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:57.385018 4914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:57.387979 4914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/master-1/data/instance:
uuid: "d44fd4dbe8224ecf93501b1d97a76304"
format_stamp: "Formatted at 2026-09-24 16:12:57 on dist-test-slave-v5j9"
server_key: "0e5d09e8077c7c827d8412a4f333c111"
server_key_iv: "f9cd622e7febaa9f1611d85a77264ac8"
server_key_version: "encryptionkey@0"
I20260924 16:12:57.388765 4914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/master-1/wal/instance:
uuid: "d44fd4dbe8224ecf93501b1d97a76304"
format_stamp: "Formatted at 2026-09-24 16:12:57 on dist-test-slave-v5j9"
server_key: "0e5d09e8077c7c827d8412a4f333c111"
server_key_iv: "f9cd622e7febaa9f1611d85a77264ac8"
server_key_version: "encryptionkey@0"
I20260924 16:12:57.393781 4914 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260924 16:12:57.396993 4929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:57.398290 4914 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:12:57.398469 4914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/master-1/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/master-1/wal
uuid: "d44fd4dbe8224ecf93501b1d97a76304"
format_stamp: "Formatted at 2026-09-24 16:12:57 on dist-test-slave-v5j9"
server_key: "0e5d09e8077c7c827d8412a4f333c111"
server_key_iv: "f9cd622e7febaa9f1611d85a77264ac8"
server_key_version: "encryptionkey@0"
I20260924 16:12:57.398664 4914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:57.422607 4914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:57.426864 4914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:57.427122 4914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:57.444667 4775 heartbeater.cc:503] Master 127.29.234.62:40099 was elected leader, sending a full tablet report...
I20260924 16:12:57.445271 4914 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.61:42395
I20260924 16:12:57.445315 4980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.61:42395 every 8 connection(s)
I20260924 16:12:57.446985 4914 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/master-1/data/info.pb
I20260924 16:12:57.450189 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 4914
I20260924 16:12:57.451213 4981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:57.460301 4981 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.234.62" port: 40099 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 16:12:57 dist-test-slave-v5j9 krb5kdc[4399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266374, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.29.234.61@KRBTEST.COM
Sep 24 16:12:57 dist-test-slave-v5j9 krb5kdc[4399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266377, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.234.61@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM
I20260924 16:12:57.475492 4984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:57.452562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52052 (local address 127.29.234.61:42395)
0924 16:12:57.453292 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:57.453304 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:57.453404 (+ 100us) server_negotiation.cc:407] Connection header received
0924 16:12:57.454958 (+ 1554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:57.454980 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:57.455458 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:57.455863 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:57.457447 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:57.458957 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:57.460426 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:57.461212 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:57.461442 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:57.466547 (+ 5105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:57.466557 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:57.466570 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:57.466614 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:57.470921 (+ 4307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:57.471571 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:57.471579 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:57.471590 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:57.471812 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:57.472258 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:57.472261 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:57.472263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:57.472801 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:57.473036 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:57.473641 (+ 605us) server_negotiation.cc:299] Negotiation successful
0924 16:12:57.474072 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":322,"threads_started":1}
I20260924 16:12:57.479453 4985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:57.462718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:40099 (local address 127.0.0.1:36986)
0924 16:12:57.463502 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:57.463550 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:57.463731 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:57.464469 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:57.464476 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:57.465328 (+ 852us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:57.465690 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:57.465710 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:57.466937 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:57.466941 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:57.468724 (+ 1783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:57.468741 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:57.469963 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:57.469969 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:57.470146 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:57.470736 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:57.470762 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:57.473269 (+ 2507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:57.476985 (+ 3716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:57.476992 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:57.476997 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:57.477476 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:57.477862 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:57.477867 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:57.477871 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:57.478047 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:57.478669 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:57.478674 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:57.478829 (+ 155us) client_negotiation.cc:769] Sending connection context
0924 16:12:57.478978 (+ 149us) client_negotiation.cc:240] Negotiation successful
0924 16:12:57.479189 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":422,"threads_started":1}
I20260924 16:12:57.479475 4986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:57.462759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36986 (local address 127.29.234.62:40099)
0924 16:12:57.464167 (+ 1408us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:57.464172 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:57.464187 (+ 15us) server_negotiation.cc:407] Connection header received
0924 16:12:57.464234 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:57.464238 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:57.464294 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:57.464413 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:57.465841 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:57.466794 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:57.468896 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:57.469593 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:57.469719 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:57.473492 (+ 3773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:57.473498 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:57.473502 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:57.473558 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:57.476811 (+ 3253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:57.477586 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:57.477594 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:57.477598 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:57.477748 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:57.478213 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:57.478217 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:57.478220 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:57.478546 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:57.478657 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:57.479087 (+ 430us) server_negotiation.cc:299] Negotiation successful
0924 16:12:57.479239 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1291,"thread_start_us":1110,"threads_started":1}
I20260924 16:12:57.480947 4981 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.234.61" port: 42395 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 16:12:57 dist-test-slave-v5j9 krb5kdc[4399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266377, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.234.61@KRBTEST.COM for oryx/127.29.234.61@KRBTEST.COM
I20260924 16:12:57.492370 4985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:57.481678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.61:42395 (local address 127.0.0.1:52066)
0924 16:12:57.481853 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:57.481881 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:57.481962 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:57.482261 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:57.482265 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:57.482548 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:57.482687 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:57.482695 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:57.483590 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:57.483594 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:57.484480 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:57.484491 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:57.485107 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:57.485111 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:57.485466 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:57.486038 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:57.486054 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:57.487917 (+ 1863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:57.490493 (+ 2576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:57.490497 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:57.490499 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:57.490865 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:57.491168 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:57.491171 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:57.491174 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:57.491290 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:57.491707 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:57.491709 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:57.491835 (+ 126us) client_negotiation.cc:769] Sending connection context
0924 16:12:57.491942 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 16:12:57.492138 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 16:12:57.492573 4984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:57.481774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52066 (local address 127.29.234.61:42395)
0924 16:12:57.481929 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:57.481933 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:57.481948 (+ 15us) server_negotiation.cc:407] Connection header received
0924 16:12:57.482076 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:57.482080 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:57.482131 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:57.482202 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:57.482807 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:57.483462 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:57.484593 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:57.484990 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:57.485073 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:57.488056 (+ 2983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:57.488060 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:57.488063 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:57.488088 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:57.490357 (+ 2269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:57.490962 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:57.490965 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:57.490968 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:57.491077 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:57.491374 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:57.491377 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:57.491378 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:57.491615 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:57.491714 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:57.492018 (+ 304us) server_negotiation.cc:299] Negotiation successful
0924 16:12:57.492260 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":60}
I20260924 16:12:57.497248 4981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304: Bootstrap starting.
I20260924 16:12:57.500376 4981 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:57.501391 4981 log.cc:824] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304: Log is configured to *not* fsync() on all Append() calls
I20260924 16:12:57.504051 4981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304: No bootstrap required, opened a new log
I20260924 16:12:57.510864 4981 raft_consensus.cc:348] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3639f056333442f859ccc6c600711ac" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 40099 } } peers { permanent_uuid: "d44fd4dbe8224ecf93501b1d97a76304" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 42395 } }
I20260924 16:12:57.511144 4981 raft_consensus.cc:374] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:57.511227 4981 raft_consensus.cc:729] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d44fd4dbe8224ecf93501b1d97a76304, State: Initialized, Role: FOLLOWER
I20260924 16:12:57.511827 4981 consensus_queue.cc:260] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3639f056333442f859ccc6c600711ac" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 40099 } } peers { permanent_uuid: "d44fd4dbe8224ecf93501b1d97a76304" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 42395 } }
I20260924 16:12:57.512591 4989 sys_catalog.cc:455] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3639f056333442f859ccc6c600711ac" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 40099 } } peers { permanent_uuid: "d44fd4dbe8224ecf93501b1d97a76304" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 42395 } } }
I20260924 16:12:57.512934 4989 sys_catalog.cc:458] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:12:57.513661 4981 sys_catalog.cc:564] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304 [sys.catalog]: configured and running, proceeding with master startup.
W20260924 16:12:57.522620 5000 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 16:12:57.522753 5000 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 16:12:57.525012 4914 master_runner.cc:428] Detected that this master d44fd4dbe8224ecf93501b1d97a76304 is joining an existing cluster
I20260924 16:12:57.525095 4914 master_runner.cc:432] Initiating AddMaster RPC to add 127.29.234.61:42395
WARNING: no policy specified for oryx/127.29.234.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.234.60@KRBTEST.COM" created.
I20260924 16:12:57.561837 5007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:57.528905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.61:42395 (local address 127.0.0.1:52080)
0924 16:12:57.541356 (+ 12451us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:57.541376 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:57.541834 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:57.542333 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:57.542340 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:57.545872 (+ 3532us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:57.546335 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:57.546354 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:57.547831 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:57.547837 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:57.549542 (+ 1705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:57.549557 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:57.550842 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:57.550848 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:57.551023 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:57.552054 (+ 1031us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:57.552095 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:57.553369 (+ 1274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:57.558170 (+ 4801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:57.558177 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:57.558181 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:57.558933 (+ 752us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:57.559515 (+ 582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:57.559519 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:57.559521 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:57.559662 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:57.560463 (+ 801us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:57.560468 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:57.560709 (+ 241us) client_negotiation.cc:769] Sending connection context
0924 16:12:57.560893 (+ 184us) client_negotiation.cc:240] Negotiation successful
0924 16:12:57.561241 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12277,"mutex_wait_us":7,"spinlock_wait_cycles":33024,"thread_start_us":12552,"threads_started":1}
I20260924 16:12:57.562310 4984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:57.529506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52080 (local address 127.29.234.61:42395)
0924 16:12:57.529692 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:57.529698 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:57.541455 (+ 11757us) server_negotiation.cc:407] Connection header received
0924 16:12:57.541978 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:57.541984 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:57.542056 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:57.542122 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:57.546516 (+ 4394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:57.547624 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:57.549698 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:57.550422 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:57.550520 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:57.553573 (+ 3053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:57.553580 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:57.553585 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:57.553618 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:57.557960 (+ 4342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:57.559119 (+ 1159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:57.559125 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:57.559128 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:57.559274 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:57.559794 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:57.559799 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:57.559803 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:57.560237 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:57.560411 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:57.561037 (+ 626us) server_negotiation.cc:299] Negotiation successful
0924 16:12:57.561457 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":115,"server-negotiator.queue_time_us":56}
I20260924 16:12:57.563966 4986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:57.529185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36990 (local address 127.29.234.62:40099)
0924 16:12:57.529382 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:57.529389 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:57.541336 (+ 11947us) server_negotiation.cc:407] Connection header received
0924 16:12:57.541953 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:57.541960 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:57.542056 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:57.542130 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:57.546570 (+ 4440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:57.547788 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:57.550113 (+ 2325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:57.550911 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:57.551021 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:57.553855 (+ 2834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:57.553864 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:57.553870 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:57.553918 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:57.557863 (+ 3945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:57.558990 (+ 1127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:57.559003 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:57.559006 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:57.559199 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:57.559789 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:57.559794 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:57.559797 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:57.560224 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:57.560388 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:57.563328 (+ 2940us) server_negotiation.cc:299] Negotiation successful
0924 16:12:57.563596 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 16:12:57.565085 5006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:57.528784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:40099 (local address 127.0.0.1:36990)
0924 16:12:57.541202 (+ 12418us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:57.541227 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:57.541772 (+ 545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:57.542362 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:57.542367 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:57.545937 (+ 3570us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:57.546382 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:57.546399 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:57.547999 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:57.548007 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:57.549928 (+ 1921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:57.549949 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:57.551332 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:57.551341 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:57.551542 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:57.552173 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:57.552196 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:57.553576 (+ 1380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:57.558020 (+ 4444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:57.558026 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:57.558030 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:57.558783 (+ 753us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:57.559321 (+ 538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:57.559326 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:57.559329 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:57.559612 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:57.560456 (+ 844us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:57.560464 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:57.562892 (+ 2428us) client_negotiation.cc:769] Sending connection context
0924 16:12:57.564156 (+ 1264us) client_negotiation.cc:240] Negotiation successful
0924 16:12:57.564625 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12230,"mutex_wait_us":2158,"thread_start_us":12603,"threads_started":1}
Sep 24 16:12:57 dist-test-slave-v5j9 krb5kdc[4399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266375, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.234.62@KRBTEST.COM for oryx/127.29.234.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.234.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/krb5kdc/oryx_127.29.234.60.keytab.
Entry for principal oryx/127.29.234.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/krb5kdc/oryx_127.29.234.60.keytab.
I20260924 16:12:57.593865 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.60:45643
--webserver_interface=127.29.234.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/krb5kdc/oryx_127.29.234.60.keytab
--principal=oryx/127.29.234.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:40243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.234.62:40099,127.29.234.61:42395,127.29.234.60:45643
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:12:57.596125 5010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:57.572785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.61:42395 (local address 127.0.0.1:52090)
0924 16:12:57.578288 (+ 5503us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:57.578342 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:57.578626 (+ 284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:57.579287 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:57.579296 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:57.580321 (+ 1025us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:57.580652 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:57.580666 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:57.582409 (+ 1743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:57.582415 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:57.584100 (+ 1685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:57.584152 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:57.585065 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:57.585072 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:57.585328 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:57.586212 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:57.586238 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:57.589336 (+ 3098us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:57.592860 (+ 3524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:57.592876 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:57.592885 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:57.593432 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:57.593823 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:57.593829 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:57.593833 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:57.594026 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:57.594813 (+ 787us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:57.594823 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:57.595046 (+ 223us) client_negotiation.cc:769] Sending connection context
0924 16:12:57.595232 (+ 186us) client_negotiation.cc:240] Negotiation successful
0924 16:12:57.595602 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5134,"thread_start_us":4944,"threads_started":1}
I20260924 16:12:57.597108 4984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:57.572982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52090 (local address 127.29.234.61:42395)
0924 16:12:57.573157 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:57.573164 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:57.578454 (+ 5290us) server_negotiation.cc:407] Connection header received
0924 16:12:57.578899 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:57.578908 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:57.579059 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:57.579211 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:57.580854 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:57.582244 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:57.584359 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:57.584840 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:57.585048 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:57.589575 (+ 4527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:57.589581 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:57.589584 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:57.589618 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:57.592703 (+ 3085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:57.593562 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:57.593567 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:57.593573 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:57.593709 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:57.594181 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:57.594188 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:57.594190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:57.594592 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:57.594725 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:57.596461 (+ 1736us) server_negotiation.cc:299] Negotiation successful
0924 16:12:57.596698 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 16:12:57.601541 4989 raft_consensus.cc:482] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:12:57.601903 4989 raft_consensus.cc:504] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3639f056333442f859ccc6c600711ac" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 40099 } } peers { permanent_uuid: "d44fd4dbe8224ecf93501b1d97a76304" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 42395 } }
I20260924 16:12:57.603506 4989 leader_election.cc:288] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3639f056333442f859ccc6c600711ac (127.29.234.62:40099)
I20260924 16:12:57.604444 4447 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.29.234.61:42395: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "d44fd4dbe8224ecf93501b1d97a76304"
member_type: NON_VOTER
last_known_addr {
host: "127.29.234.61"
port: 42395
}
attrs {
promote: true
}
}
dest_uuid: "d3639f056333442f859ccc6c600711ac"
cas_config_opid_index: -1
I20260924 16:12:57.604468 4457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d44fd4dbe8224ecf93501b1d97a76304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3639f056333442f859ccc6c600711ac" is_pre_election: true
I20260924 16:12:57.605664 4932 leader_election.cc:302] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: d44fd4dbe8224ecf93501b1d97a76304; no voters: d3639f056333442f859ccc6c600711ac
I20260924 16:12:57.605747 4447 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3639f056333442f859ccc6c600711ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d3639f056333442f859ccc6c600711ac" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 40099 } } peers { permanent_uuid: "d44fd4dbe8224ecf93501b1d97a76304" member_type: NON_VOTER last_known_addr { host: "127.29.234.61" port: 42395 } attrs { promote: true } }
I20260924 16:12:57.606406 4989 raft_consensus.cc:2726] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 16:12:57.627048 5013 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d3639f056333442f859ccc6c600711ac [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER d44fd4dbe8224ecf93501b1d97a76304 (127.29.234.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d3639f056333442f859ccc6c600711ac" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 40099 } } peers { permanent_uuid: "d44fd4dbe8224ecf93501b1d97a76304" member_type: NON_VOTER last_known_addr { host: "127.29.234.61" port: 42395 } attrs { promote: true } } }
I20260924 16:12:57.628085 4956 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:57.632335 4956 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304 [term 1 FOLLOWER]: Refusing update from remote peer d3639f056333442f859ccc6c600711ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260924 16:12:57.632616 5013 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.29.234.61:42395
I20260924 16:12:57.633756 5015 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d3639f056333442f859ccc6c600711ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "d44fd4dbe8224ecf93501b1d97a76304" member_type: NON_VOTER last_known_addr { host: "127.29.234.61" port: 42395 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:12:57.634519 4914 master.cc:537] Master@127.29.234.61:42395 shutting down...
W20260924 16:12:57.637634 4431 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260924 16:12:57.638105 5014 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3639f056333442f859ccc6c600711ac [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d3639f056333442f859ccc6c600711ac" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d3639f056333442f859ccc6c600711ac" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 40099 } } peers { permanent_uuid: "d44fd4dbe8224ecf93501b1d97a76304" member_type: NON_VOTER last_known_addr { host: "127.29.234.61" port: 42395 } attrs { promote: true } } }
I20260924 16:12:57.641788 5014 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3639f056333442f859ccc6c600711ac [sys.catalog]: This master's current role is: LEADER
I20260924 16:12:57.652402 4914 raft_consensus.cc:2229] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:12:57.652987 4914 raft_consensus.cc:2258] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:12:57.653268 4914 tablet_replica.cc:330] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304: stopping tablet replica
W20260924 16:12:57.657423 4431 consensus_peers.cc:487] T 00000000000000000000000000000000 P d3639f056333442f859ccc6c600711ac -> Peer d44fd4dbe8224ecf93501b1d97a76304 (127.29.234.61:42395): Couldn't send request to peer d44fd4dbe8224ecf93501b1d97a76304. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 16:12:57.661441 4914 master.cc:552] Master@127.29.234.61:42395 shutdown complete.
I20260924 16:12:57.661609 4914 master_runner.cc:305] Clearing existing system tablet
I20260924 16:12:57.662070 4914 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:57.662182 4914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:57.670096 4914 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.004s sys 0.004s
I20260924 16:12:57.673705 5020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:57.674170 4914 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:57.674758 4914 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260924 16:12:57.674943 4914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/master-1/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/master-1/wal
uuid: "d44fd4dbe8224ecf93501b1d97a76304"
format_stamp: "Formatted at 2026-09-24 16:12:57 on dist-test-slave-v5j9"
server_key: "0e5d09e8077c7c827d8412a4f333c111"
server_key_iv: "f9cd622e7febaa9f1611d85a77264ac8"
server_key_version: "encryptionkey@0"
I20260924 16:12:57.680377 4914 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:12:57.687652 4914 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 16:12:57.687968 4914 log.cc:1196] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304: Deleting WAL directory at /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260924 16:12:57.688547 4914 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304: Deleting consensus metadata
I20260924 16:12:57.688992 4914 master_runner.cc:315] Copying system tablet from 127.29.234.62:40099
I20260924 16:12:57.691737 4914 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304: tablet copy: Beginning tablet copy session from remote peer at address 127.29.234.62:40099
I20260924 16:12:57.708503 4986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:57.692724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37004 (local address 127.29.234.62:40099)
0924 16:12:57.692910 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:57.692917 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:57.693959 (+ 1042us) server_negotiation.cc:407] Connection header received
0924 16:12:57.694288 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:57.694294 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:57.694393 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:57.694469 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:57.695694 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:57.696817 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:57.698764 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:57.699378 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:57.699512 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:57.701842 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:57.701849 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:57.701853 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:57.701892 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:57.705390 (+ 3498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:57.706258 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:57.706263 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:57.706267 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:57.706433 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:57.706879 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:57.706884 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:57.706887 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:57.707248 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:57.707412 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:57.707884 (+ 472us) server_negotiation.cc:299] Negotiation successful
0924 16:12:57.708112 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 16:12:57.708544 5027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:57.692684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:40099 (local address 127.0.0.1:37004)
0924 16:12:57.693838 (+ 1154us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:57.693861 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:57.694084 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:57.694693 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:57.694698 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:57.695215 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:57.695513 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:57.695527 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:57.696959 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:57.696965 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:57.698590 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:57.698607 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:57.699549 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:57.699555 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:57.699754 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:57.700526 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:57.700548 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:57.701641 (+ 1093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:57.705550 (+ 3909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:57.705556 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:57.705560 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:57.706083 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:57.706553 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:57.706558 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:57.706562 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:57.706748 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:57.707378 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:57.707383 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:57.707566 (+ 183us) client_negotiation.cc:769] Sending connection context
0924 16:12:57.707733 (+ 167us) client_negotiation.cc:240] Negotiation successful
0924 16:12:57.708076 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":949,"thread_start_us":808,"threads_started":1}
I20260924 16:12:57.709826 4467 tablet_copy_service.cc:140] P d3639f056333442f859ccc6c600711ac: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer d44fd4dbe8224ecf93501b1d97a76304 ({username='oryx', principal='oryx/127.29.234.61@KRBTEST.COM'} at 127.0.0.1:37004)
I20260924 16:12:57.710084 4467 tablet_copy_service.cc:161] P d3639f056333442f859ccc6c600711ac: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer d44fd4dbe8224ecf93501b1d97a76304 at {username='oryx', principal='oryx/127.29.234.61@KRBTEST.COM'} at 127.0.0.1:37004: session id = d44fd4dbe8224ecf93501b1d97a76304-00000000000000000000000000000000
I20260924 16:12:57.715255 4467 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P d3639f056333442f859ccc6c600711ac: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 16:12:57.717702 4914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 16:12:57.724624 4914 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304: tablet copy: Starting download of 0 data blocks...
I20260924 16:12:57.724948 4914 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304: tablet copy: Starting download of 1 WAL segments...
I20260924 16:12:57.728185 4914 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 16:12:57.732564 4467 tablet_copy_service.cc:342] P d3639f056333442f859ccc6c600711ac: Request end of tablet copy session d44fd4dbe8224ecf93501b1d97a76304-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.29.234.61@KRBTEST.COM'} at 127.0.0.1:37004
I20260924 16:12:57.732755 4467 tablet_copy_service.cc:433] P d3639f056333442f859ccc6c600711ac: ending tablet copy session d44fd4dbe8224ecf93501b1d97a76304-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer d44fd4dbe8224ecf93501b1d97a76304
I20260924 16:12:57.744822 4914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:57.747884 5028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:57.748446 5029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:57.755832 5032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:57.758852 4914 server_base.cc:1034] running on GCE node
I20260924 16:12:57.759567 4914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:57.760378 4914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:57.761619 4914 hybrid_clock.cc:648] HybridClock initialized: now 1790266377761528 us; error 53 us; skew 500 ppm
I20260924 16:12:57.761916 4914 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:12:57 dist-test-slave-v5j9 krb5kdc[4399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266377, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.234.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:57.764176 4914 init.cc:377] Logged in from keytab as oryx/127.29.234.61@KRBTEST.COM (short username oryx)
I20260924 16:12:57.769250 4914 webserver.cc:460] Webserver started at http://127.29.234.61:46789/ using document root <none> and password file <none>
I20260924 16:12:57.769680 4914 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:57.769773 4914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:57.772985 4914 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260924 16:12:57.774924 5039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:57.775421 4914 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 16:12:57.775542 4914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/master-1/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/master-1/wal
uuid: "d44fd4dbe8224ecf93501b1d97a76304"
format_stamp: "Formatted at 2026-09-24 16:12:57 on dist-test-slave-v5j9"
server_key: "0e5d09e8077c7c827d8412a4f333c111"
server_key_iv: "f9cd622e7febaa9f1611d85a77264ac8"
server_key_version: "encryptionkey@0"
I20260924 16:12:57.775707 4914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:57.800868 4914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:57.805281 4914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:57.828001 4914 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.61:42395
I20260924 16:12:57.828055 5091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.61:42395 every 8 connection(s)
I20260924 16:12:57.829087 4914 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/master-1/data/info.pb
I20260924 16:12:57.834863 5092 sys_catalog.cc:263] Verifying existing consensus state
I20260924 16:12:57.840710 5092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304: Bootstrap starting.
I20260924 16:12:57.860407 5092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 16:12:57.860983 5092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304: Bootstrap complete.
I20260924 16:12:57.862164 5092 raft_consensus.cc:348] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d3639f056333442f859ccc6c600711ac" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 40099 } } peers { permanent_uuid: "d44fd4dbe8224ecf93501b1d97a76304" member_type: NON_VOTER last_known_addr { host: "127.29.234.61" port: 42395 } attrs { promote: true } }
I20260924 16:12:57.862454 5092 raft_consensus.cc:729] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d44fd4dbe8224ecf93501b1d97a76304, State: Initialized, Role: LEARNER
I20260924 16:12:57.862733 5092 consensus_queue.cc:260] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d3639f056333442f859ccc6c600711ac" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 40099 } } peers { permanent_uuid: "d44fd4dbe8224ecf93501b1d97a76304" member_type: NON_VOTER last_known_addr { host: "127.29.234.61" port: 42395 } attrs { promote: true } }
I20260924 16:12:57.863690 5095 sys_catalog.cc:455] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d3639f056333442f859ccc6c600711ac" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 40099 } } peers { permanent_uuid: "d44fd4dbe8224ecf93501b1d97a76304" member_type: NON_VOTER last_known_addr { host: "127.29.234.61" port: 42395 } attrs { promote: true } } }
I20260924 16:12:57.864125 5095 sys_catalog.cc:458] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304 [sys.catalog]: This master's current role is: LEARNER
I20260924 16:12:57.864262 5092 sys_catalog.cc:564] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:12:57.875912 5106 catalog_manager.cc:1259] Loaded cluster ID: e95643b1533745a68073f27a96aa4b1a
I20260924 16:12:57.876093 5106 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304: loading cluster ID for follower catalog manager: success
I20260924 16:12:57.881387 5106 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304: acquiring CA information for follower catalog manager: success
I20260924 16:12:57.885010 5106 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 16:12:57.958081 4909 heartbeater.cc:503] Master 127.29.234.62:40099 was elected leader, sending a full tablet report...
W20260924 16:12:58.043555 5011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:58.043900 5011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:58.044057 5011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:58.051833 5011 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:12:58.052078 5011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:58.052246 5011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:58.052448 5011 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:12:58.052603 5011 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:12:58.065042 5011 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:40243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.29.234.62:40099,127.29.234.61:42395,127.29.234.60:45643
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/krb5kdc/oryx_127.29.234.60.keytab
--principal=oryx/127.29.234.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.60:45643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.29.234.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:12:58.066709 5011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:58.068224 5011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:58.080495 5111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:58.080420 5114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:58.080456 5112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:58.081851 5011 server_base.cc:1034] running on GCE node
I20260924 16:12:58.082918 5011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:58.084919 5011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:58.086277 5011 hybrid_clock.cc:648] HybridClock initialized: now 1790266378086212 us; error 49 us; skew 500 ppm
I20260924 16:12:58.086654 5011 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:12:58 dist-test-slave-v5j9 krb5kdc[4399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266378, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.234.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:58.090464 5116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:58.082907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52098 (local address 127.29.234.61:42395)
0924 16:12:58.083478 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:58.083484 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:58.083509 (+ 25us) server_negotiation.cc:407] Connection header received
0924 16:12:58.083598 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:58.083602 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:58.083710 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:58.083844 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:12:58.084394 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:58.085794 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:58.088031 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:58.089335 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:58.089525 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:58.089746 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:58.090122 (+ 376us) server_negotiation.cc:299] Negotiation successful
0924 16:12:58.090223 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":274,"threads_started":1}
I20260924 16:12:58.090512 5010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:58.082648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.61:42395 (local address 127.0.0.1:52098)
0924 16:12:58.082890 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:58.082909 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:58.083045 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:58.083984 (+ 939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:58.083989 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:58.084032 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:12:58.084216 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:58.084227 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:58.085934 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:58.085939 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:58.087873 (+ 1934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:58.087888 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:58.089555 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:58.089562 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:58.089841 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:58.089910 (+ 69us) client_negotiation.cc:769] Sending connection context
0924 16:12:58.090025 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 16:12:58.090248 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260924 16:12:58.090708 5011 init.cc:377] Logged in from keytab as oryx/127.29.234.60@KRBTEST.COM (short username oryx)
I20260924 16:12:58.091801 5067 raft_consensus.cc:1203] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260924 16:12:58.092501 5095 sys_catalog.cc:455] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3639f056333442f859ccc6c600711ac. Latest consensus state: current_term: 1 leader_uuid: "d3639f056333442f859ccc6c600711ac" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d3639f056333442f859ccc6c600711ac" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 40099 } } peers { permanent_uuid: "d44fd4dbe8224ecf93501b1d97a76304" member_type: NON_VOTER last_known_addr { host: "127.29.234.61" port: 42395 } attrs { promote: true } } }
I20260924 16:12:58.092759 5011 webserver.cc:460] Webserver started at http://127.29.234.60:44713/ using document root <none> and password file <none>
I20260924 16:12:58.092816 5095 sys_catalog.cc:458] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304 [sys.catalog]: This master's current role is: LEARNER
I20260924 16:12:58.093555 5011 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:58.093709 5011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:58.094064 5011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:12:58.097034 5011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/master-2/data/instance:
uuid: "7785d6b6f5bf4dd5a2667857b24dcf8a"
format_stamp: "Formatted at 2026-09-24 16:12:58 on dist-test-slave-v5j9"
server_key: "9ae97c93f3b3079faa734ab973547532"
server_key_iv: "4992c6a783253935974a68f5ec6d2e19"
server_key_version: "encryptionkey@0"
I20260924 16:12:58.097816 5011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/master-2/wal/instance:
uuid: "7785d6b6f5bf4dd5a2667857b24dcf8a"
format_stamp: "Formatted at 2026-09-24 16:12:58 on dist-test-slave-v5j9"
server_key: "9ae97c93f3b3079faa734ab973547532"
server_key_iv: "4992c6a783253935974a68f5ec6d2e19"
server_key_version: "encryptionkey@0"
I20260924 16:12:58.102957 5011 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260924 16:12:58.106760 5122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:58.108340 5011 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:12:58.108508 5011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/master-2/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/master-2/wal
uuid: "7785d6b6f5bf4dd5a2667857b24dcf8a"
format_stamp: "Formatted at 2026-09-24 16:12:58 on dist-test-slave-v5j9"
server_key: "9ae97c93f3b3079faa734ab973547532"
server_key_iv: "4992c6a783253935974a68f5ec6d2e19"
server_key_version: "encryptionkey@0"
I20260924 16:12:58.108701 5011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:58.131392 5011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:58.136070 5011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:58.136341 5011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:58.155457 5011 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.60:45643
I20260924 16:12:58.155328 5175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.60:45643 every 8 connection(s)
I20260924 16:12:58.156956 5011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/master-2/data/info.pb
I20260924 16:12:58.161217 5176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:12:58.165828 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 5011
I20260924 16:12:58.169241 5176 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.234.62" port: 40099 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 16:12:58 dist-test-slave-v5j9 krb5kdc[4399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266378, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.234.60@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM
Sep 24 16:12:58 dist-test-slave-v5j9 krb5kdc[4399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266374, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.29.234.60@KRBTEST.COM
I20260924 16:12:58.188866 4986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:58.173134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37016 (local address 127.29.234.62:40099)
0924 16:12:58.173412 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:58.173421 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:58.174034 (+ 613us) server_negotiation.cc:407] Connection header received
0924 16:12:58.174250 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:58.174257 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:58.174324 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:58.174428 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:58.175635 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:58.176354 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:58.178185 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:58.178751 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:58.178840 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:58.182763 (+ 3923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:58.182769 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:58.182780 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:58.182811 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:58.185731 (+ 2920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:58.186541 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:58.186546 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:58.186554 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:58.186676 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:58.187153 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:58.187156 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:58.187159 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:58.187420 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:58.187574 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:58.188280 (+ 706us) server_negotiation.cc:299] Negotiation successful
0924 16:12:58.188510 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 16:12:58.189996 5179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:58.168184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35452 (local address 127.29.234.60:45643)
0924 16:12:58.169018 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:58.169031 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:58.169071 (+ 40us) server_negotiation.cc:407] Connection header received
0924 16:12:58.171196 (+ 2125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:58.171221 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:58.171679 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:58.172059 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:58.173795 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:58.175230 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:58.176958 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:58.177662 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:58.177794 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:58.183237 (+ 5443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:58.183245 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:58.183258 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:58.183289 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:58.186217 (+ 2928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:58.186932 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:58.186939 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:58.186942 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:58.187080 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:58.187454 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:58.187459 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:58.187462 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:58.187939 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:58.188277 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:58.188580 (+ 303us) server_negotiation.cc:299] Negotiation successful
0924 16:12:58.188825 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"spinlock_wait_cycles":15232,"thread_start_us":421,"threads_started":1}
I20260924 16:12:58.190033 5180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:58.173001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:40099 (local address 127.0.0.1:37016)
0924 16:12:58.173904 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:58.173938 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:58.174123 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:58.174502 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:58.174509 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:58.175141 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:58.175474 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:58.175501 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:58.176491 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:58.176497 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:58.178049 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:58.178063 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:58.178973 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:58.178981 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:58.179169 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:58.179799 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:58.179833 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:58.182563 (+ 2730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:58.185871 (+ 3308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:58.185887 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:58.185902 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:58.186412 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:58.186792 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:58.186798 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:58.186801 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:58.187026 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:58.187557 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:58.187565 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:58.187921 (+ 356us) client_negotiation.cc:769] Sending connection context
0924 16:12:58.188185 (+ 264us) client_negotiation.cc:240] Negotiation successful
0924 16:12:58.188687 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"mutex_wait_us":1,"spinlock_wait_cycles":17152,"thread_start_us":527,"threads_started":1}
I20260924 16:12:58.191975 5176 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.234.61" port: 42395 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 16:12:58 dist-test-slave-v5j9 krb5kdc[4399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266378, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.234.60@KRBTEST.COM for oryx/127.29.234.61@KRBTEST.COM
I20260924 16:12:58.205212 5116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:58.192767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52112 (local address 127.29.234.61:42395)
0924 16:12:58.192962 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:58.192968 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:58.193034 (+ 66us) server_negotiation.cc:407] Connection header received
0924 16:12:58.193192 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:58.193198 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:58.193278 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:58.193414 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:58.194007 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:58.195043 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:58.196227 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:58.196888 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:58.196993 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:58.200087 (+ 3094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:58.200093 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:58.200096 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:58.200127 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:58.202822 (+ 2695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:58.203636 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:58.203639 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:58.203641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:58.203758 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:58.204132 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:58.204135 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:58.204137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:58.204369 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:58.204470 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:58.204800 (+ 330us) server_negotiation.cc:299] Negotiation successful
0924 16:12:58.204987 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 16:12:58.205482 5180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:58.192743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.61:42395 (local address 127.0.0.1:52112)
0924 16:12:58.192940 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:58.192963 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:58.193069 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:58.193445 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:58.193450 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:58.193755 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:58.193888 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:58.193895 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:58.195150 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:58.195153 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:58.196086 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:58.196096 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:58.197012 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:58.197015 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:58.197139 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:58.197719 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:58.197735 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:58.199887 (+ 2152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:58.202952 (+ 3065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:58.202956 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:58.202959 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:58.203541 (+ 582us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:58.203847 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:58.203850 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:58.203852 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:58.204043 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:58.204461 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:58.204464 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:58.204607 (+ 143us) client_negotiation.cc:769] Sending connection context
0924 16:12:58.204715 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 16:12:58.204988 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260924 16:12:58.206519 5176 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.234.60" port: 45643 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 16:12:58 dist-test-slave-v5j9 krb5kdc[4399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266378, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.234.60@KRBTEST.COM for oryx/127.29.234.60@KRBTEST.COM
I20260924 16:12:58.218192 5180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:58.207436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.60:45643 (local address 127.0.0.1:35456)
0924 16:12:58.207568 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:58.207582 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:58.207689 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:58.207958 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:58.207962 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:58.208303 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:58.208435 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:58.208442 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:58.209391 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:58.209400 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:58.210178 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:58.210188 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:58.210840 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:58.210845 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:58.211072 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:58.211514 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:58.211529 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:58.213425 (+ 1896us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:58.216082 (+ 2657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:58.216085 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:58.216088 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:58.216445 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:58.216742 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:58.216746 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:58.216748 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:58.216906 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:58.217451 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:58.217455 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:58.217569 (+ 114us) client_negotiation.cc:769] Sending connection context
0924 16:12:58.217708 (+ 139us) client_negotiation.cc:240] Negotiation successful
0924 16:12:58.217895 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":3}
I20260924 16:12:58.218313 5179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:58.207589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35456 (local address 127.29.234.60:45643)
0924 16:12:58.207729 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:58.207736 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:58.207753 (+ 17us) server_negotiation.cc:407] Connection header received
0924 16:12:58.207793 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:58.207804 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:58.207850 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:58.207944 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:58.208562 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:58.209231 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:58.210317 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:58.210699 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:58.210833 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:58.213630 (+ 2797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:58.213636 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:58.213641 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:58.213676 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:58.215957 (+ 2281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:58.216543 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:58.216547 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:58.216549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:58.216654 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:58.216995 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:58.216998 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:58.217000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:58.217248 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:58.217417 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:58.217790 (+ 373us) server_negotiation.cc:299] Negotiation successful
0924 16:12:58.218023 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":104,"server-negotiator.queue_time_us":50}
I20260924 16:12:58.223049 5176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7785d6b6f5bf4dd5a2667857b24dcf8a: Bootstrap starting.
I20260924 16:12:58.226330 5176 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7785d6b6f5bf4dd5a2667857b24dcf8a: Neither blocks nor log segments found. Creating new log.
I20260924 16:12:58.227362 5176 log.cc:824] T 00000000000000000000000000000000 P 7785d6b6f5bf4dd5a2667857b24dcf8a: Log is configured to *not* fsync() on all Append() calls
I20260924 16:12:58.229983 5176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7785d6b6f5bf4dd5a2667857b24dcf8a: No bootstrap required, opened a new log
I20260924 16:12:58.237077 5176 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7785d6b6f5bf4dd5a2667857b24dcf8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3639f056333442f859ccc6c600711ac" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 40099 } } peers { permanent_uuid: "d44fd4dbe8224ecf93501b1d97a76304" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 42395 } } peers { permanent_uuid: "7785d6b6f5bf4dd5a2667857b24dcf8a" member_type: VOTER last_known_addr { host: "127.29.234.60" port: 45643 } }
I20260924 16:12:58.237455 5176 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7785d6b6f5bf4dd5a2667857b24dcf8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:12:58.237586 5176 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7785d6b6f5bf4dd5a2667857b24dcf8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7785d6b6f5bf4dd5a2667857b24dcf8a, State: Initialized, Role: FOLLOWER
I20260924 16:12:58.238144 5176 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7785d6b6f5bf4dd5a2667857b24dcf8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3639f056333442f859ccc6c600711ac" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 40099 } } peers { permanent_uuid: "d44fd4dbe8224ecf93501b1d97a76304" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 42395 } } peers { permanent_uuid: "7785d6b6f5bf4dd5a2667857b24dcf8a" member_type: VOTER last_known_addr { host: "127.29.234.60" port: 45643 } }
I20260924 16:12:58.239283 5182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7785d6b6f5bf4dd5a2667857b24dcf8a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3639f056333442f859ccc6c600711ac" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 40099 } } peers { permanent_uuid: "d44fd4dbe8224ecf93501b1d97a76304" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 42395 } } peers { permanent_uuid: "7785d6b6f5bf4dd5a2667857b24dcf8a" member_type: VOTER last_known_addr { host: "127.29.234.60" port: 45643 } } }
I20260924 16:12:58.239878 5182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7785d6b6f5bf4dd5a2667857b24dcf8a [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:12:58.240103 5176 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7785d6b6f5bf4dd5a2667857b24dcf8a [sys.catalog]: configured and running, proceeding with master startup.
W20260924 16:12:58.250660 5193 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7785d6b6f5bf4dd5a2667857b24dcf8a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 16:12:58.250818 5193 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 16:12:58.255682 5011 master_runner.cc:428] Detected that this master 7785d6b6f5bf4dd5a2667857b24dcf8a is joining an existing cluster
I20260924 16:12:58.255815 5011 master_runner.cc:432] Initiating AddMaster RPC to add 127.29.234.60:45643
I20260924 16:12:58.286113 5116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:58.260692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52118 (local address 127.29.234.61:42395)
0924 16:12:58.260897 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:58.260905 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:58.269842 (+ 8937us) server_negotiation.cc:407] Connection header received
0924 16:12:58.270033 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:58.270039 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:58.270128 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:58.270232 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:58.271925 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:58.273025 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:58.275147 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:58.275774 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:58.275897 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:58.278794 (+ 2897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:58.278802 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:58.278807 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:58.278846 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:58.282797 (+ 3951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:58.283653 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:58.283659 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:58.283663 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:58.283814 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:58.284301 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:58.284307 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:58.284310 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:58.284667 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:58.284803 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:58.285472 (+ 669us) server_negotiation.cc:299] Negotiation successful
0924 16:12:58.285728 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 16:12:58.286559 5179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:58.260488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35470 (local address 127.29.234.60:45643)
0924 16:12:58.260674 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:58.260681 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:58.269900 (+ 9219us) server_negotiation.cc:407] Connection header received
0924 16:12:58.270080 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:58.270088 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:58.270160 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:58.270221 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:58.272366 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:58.273525 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:58.275828 (+ 2303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:58.276717 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:58.276828 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:58.279585 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:58.279594 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:58.279600 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:58.279643 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:58.283436 (+ 3793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:58.284203 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:58.284209 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:58.284213 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:58.284375 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:58.284787 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:58.284791 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:58.284794 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:58.285188 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:58.285311 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:58.285906 (+ 595us) server_negotiation.cc:299] Negotiation successful
0924 16:12:58.286134 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":43}
I20260924 16:12:58.287411 5199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:58.261089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.61:42395 (local address 127.0.0.1:52118)
0924 16:12:58.269713 (+ 8624us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:58.269732 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:58.269891 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:58.270414 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:58.270419 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:58.271355 (+ 936us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:58.271720 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:58.271732 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:58.273204 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:58.273209 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:58.274983 (+ 1774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:58.274999 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:58.276287 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:58.276295 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:58.276610 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:58.277304 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:58.277343 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:58.278585 (+ 1242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:58.282975 (+ 4390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:58.282980 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:58.282984 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:58.283504 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:58.283981 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:58.283986 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:58.283990 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:58.284174 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:58.284903 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:58.284908 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:58.285149 (+ 241us) client_negotiation.cc:769] Sending connection context
0924 16:12:58.286521 (+ 1372us) client_negotiation.cc:240] Negotiation successful
0924 16:12:58.286955 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8058,"mutex_wait_us":33,"spinlock_wait_cycles":9600,"thread_start_us":8345,"threads_started":1}
I20260924 16:12:58.287688 4986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:58.261061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37028 (local address 127.29.234.62:40099)
0924 16:12:58.261290 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:58.261298 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:58.269572 (+ 8274us) server_negotiation.cc:407] Connection header received
0924 16:12:58.270195 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:58.270204 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:58.270281 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:58.270349 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:58.272385 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:58.273502 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:58.276686 (+ 3184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:58.277428 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:58.277546 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:58.280615 (+ 3069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:58.280623 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:58.280628 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:58.280669 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:58.284462 (+ 3793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:58.285235 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:58.285241 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:58.285245 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:58.285437 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:58.285899 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:58.285904 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:58.285907 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:58.286285 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:58.286406 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:58.287079 (+ 673us) server_negotiation.cc:299] Negotiation successful
0924 16:12:58.287342 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 16:12:58.288277 5201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:58.260540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:40099 (local address 127.0.0.1:37028)
0924 16:12:58.269435 (+ 8895us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:58.269462 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:58.270050 (+ 588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:58.270563 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:58.270570 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:58.271532 (+ 962us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:58.272215 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:58.272223 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:58.273709 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:58.273715 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:58.276520 (+ 2805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:58.276535 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:58.277908 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:58.277914 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:58.278139 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:58.278805 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:58.278829 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:58.280379 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:58.284599 (+ 4220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:58.284603 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:58.284606 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:58.285089 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:58.285580 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:58.285585 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:58.285588 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:58.285777 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:58.286501 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:58.286510 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:58.286799 (+ 289us) client_negotiation.cc:769] Sending connection context
0924 16:12:58.286944 (+ 145us) client_negotiation.cc:240] Negotiation successful
0924 16:12:58.287658 (+ 714us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8422,"mutex_wait_us":192,"spinlock_wait_cycles":2432,"thread_start_us":8381,"threads_started":1}
I20260924 16:12:58.288820 5200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:58.261150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.60:45643 (local address 127.0.0.1:35470)
0924 16:12:58.269828 (+ 8678us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:58.269842 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:58.269968 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:58.270429 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:58.270433 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:58.271532 (+ 1099us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:58.272202 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:58.272214 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:58.273732 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:58.273738 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:58.275690 (+ 1952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:58.275704 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:58.277069 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:58.277075 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:58.277234 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:58.277918 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:58.277942 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:58.279369 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:58.283581 (+ 4212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:58.283586 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:58.283590 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:58.284072 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:58.284506 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:58.284513 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:58.284519 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:58.284666 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:58.285430 (+ 764us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:58.285435 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:58.285639 (+ 204us) client_negotiation.cc:769] Sending connection context
0924 16:12:58.286744 (+ 1105us) client_negotiation.cc:240] Negotiation successful
0924 16:12:58.288156 (+ 1412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8514,"mutex_wait_us":1198,"spinlock_wait_cycles":768,"thread_start_us":6979,"threads_started":1}
I20260924 16:12:58.296422 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 4508
Sep 24 16:12:58 dist-test-slave-v5j9 krb5kdc[4399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266375, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.234.62@KRBTEST.COM for oryx/127.29.234.60@KRBTEST.COM
Sep 24 16:12:58 dist-test-slave-v5j9 krb5kdc[4399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266374, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.29.234.1@KRBTEST.COM
I20260924 16:12:58.309561 5010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:58.293621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.60:45643 (local address 127.0.0.1:35476)
0924 16:12:58.293905 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:58.293924 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:58.294038 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:58.294392 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:58.294397 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:58.294870 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:58.295062 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:58.295074 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:58.296125 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:58.296132 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:58.297592 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:58.297608 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:58.298842 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:58.298848 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:58.299043 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:58.299666 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:58.299690 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:58.302456 (+ 2766us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:58.306351 (+ 3895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:58.306357 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:58.306363 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:58.306877 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:58.307303 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:58.307308 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:58.307312 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:58.307467 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:58.308411 (+ 944us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:58.308416 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:58.308597 (+ 181us) client_negotiation.cc:769] Sending connection context
0924 16:12:58.308778 (+ 181us) client_negotiation.cc:240] Negotiation successful
0924 16:12:58.309087 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133}
I20260924 16:12:58.309588 5179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:58.293624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35476 (local address 127.29.234.60:45643)
0924 16:12:58.293795 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:58.293799 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:58.293993 (+ 194us) server_negotiation.cc:407] Connection header received
0924 16:12:58.294168 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:58.294172 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:58.294256 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:58.294335 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:58.295242 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:58.295995 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:58.297760 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:58.298515 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:58.298648 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:58.302685 (+ 4037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:58.302690 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:58.302693 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:58.302722 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:58.306204 (+ 3482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:58.307009 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:58.307015 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:58.307019 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:58.307165 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:58.307586 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:58.307591 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:58.307595 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:58.307946 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:58.308108 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:58.308926 (+ 818us) server_negotiation.cc:299] Negotiation successful
0924 16:12:58.309161 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 16:12:58.311203 4447 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.29.234.60:45643: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "7785d6b6f5bf4dd5a2667857b24dcf8a"
member_type: NON_VOTER
last_known_addr {
host: "127.29.234.60"
port: 45643
}
attrs {
promote: true
}
}
dest_uuid: "d3639f056333442f859ccc6c600711ac"
cas_config_opid_index: 5
I20260924 16:12:58.312198 4447 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3639f056333442f859ccc6c600711ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d3639f056333442f859ccc6c600711ac" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 40099 } } peers { permanent_uuid: "d44fd4dbe8224ecf93501b1d97a76304" member_type: NON_VOTER last_known_addr { host: "127.29.234.61" port: 42395 } attrs { promote: true } } peers { permanent_uuid: "7785d6b6f5bf4dd5a2667857b24dcf8a" member_type: NON_VOTER last_known_addr { host: "127.29.234.60" port: 45643 } attrs { promote: true } }
I20260924 16:12:58.314128 5014 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d3639f056333442f859ccc6c600711ac [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 7785d6b6f5bf4dd5a2667857b24dcf8a (127.29.234.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d3639f056333442f859ccc6c600711ac" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 40099 } } peers { permanent_uuid: "d44fd4dbe8224ecf93501b1d97a76304" member_type: NON_VOTER last_known_addr { host: "127.29.234.61" port: 42395 } attrs { promote: true } } peers { permanent_uuid: "7785d6b6f5bf4dd5a2667857b24dcf8a" member_type: NON_VOTER last_known_addr { host: "127.29.234.60" port: 45643 } attrs { promote: true } } }
I20260924 16:12:58.318186 5067 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304 [term 1 LEARNER]: Refusing update from remote peer d3639f056333442f859ccc6c600711ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 16:12:58.319146 5014 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.29.234.60:45643
I20260924 16:12:58.319833 5151 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7785d6b6f5bf4dd5a2667857b24dcf8a [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:12:58.324941 5151 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7785d6b6f5bf4dd5a2667857b24dcf8a [term 1 FOLLOWER]: Refusing update from remote peer d3639f056333442f859ccc6c600711ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 16:12:58.326251 5011 master.cc:537] Master@127.29.234.60:45643 shutting down...
I20260924 16:12:58.329386 5204 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d3639f056333442f859ccc6c600711ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "d44fd4dbe8224ecf93501b1d97a76304" member_type: NON_VOTER last_known_addr { host: "127.29.234.61" port: 42395 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 16:12:58.341583 5011 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 7785d6b6f5bf4dd5a2667857b24dcf8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:12:58.342046 5011 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 7785d6b6f5bf4dd5a2667857b24dcf8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:12:58.342334 5011 tablet_replica.cc:330] T 00000000000000000000000000000000 P 7785d6b6f5bf4dd5a2667857b24dcf8a: stopping tablet replica
I20260924 16:12:58.343063 5202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:58.297529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50286 (local address 127.29.234.1:40063)
0924 16:12:58.298096 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:58.298106 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:58.298137 (+ 31us) server_negotiation.cc:407] Connection header received
0924 16:12:58.298204 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:58.298212 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:58.298467 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:58.298639 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:58.300105 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:58.301265 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:58.303292 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:58.303935 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:58.304078 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:58.309014 (+ 4936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:58.309029 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:58.309050 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:58.309094 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:58.331263 (+ 22169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:58.339762 (+ 8499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:58.339779 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:58.339786 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:58.340049 (+ 263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:58.340972 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:58.340978 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:58.340981 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:58.341527 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:58.341863 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:58.342349 (+ 486us) server_negotiation.cc:299] Negotiation successful
0924 16:12:58.342652 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":549,"threads_started":1}
I20260924 16:12:58.344751 4632 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:12:58.352784 5011 master.cc:552] Master@127.29.234.60:45643 shutdown complete.
I20260924 16:12:58.354210 5011 master_runner.cc:305] Clearing existing system tablet
I20260924 16:12:58.355676 5067 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 7785d6b6f5bf4dd5a2667857b24dcf8a (127.29.234.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d3639f056333442f859ccc6c600711ac" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 40099 } } peers { permanent_uuid: "d44fd4dbe8224ecf93501b1d97a76304" member_type: NON_VOTER last_known_addr { host: "127.29.234.61" port: 42395 } attrs { promote: true } } peers { permanent_uuid: "7785d6b6f5bf4dd5a2667857b24dcf8a" member_type: NON_VOTER last_known_addr { host: "127.29.234.60" port: 45643 } attrs { promote: true } } }
I20260924 16:12:58.361963 5011 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:58.362147 5011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:58.367596 5095 sys_catalog.cc:455] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d3639f056333442f859ccc6c600711ac" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d3639f056333442f859ccc6c600711ac" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 40099 } } peers { permanent_uuid: "d44fd4dbe8224ecf93501b1d97a76304" member_type: NON_VOTER last_known_addr { host: "127.29.234.61" port: 42395 } attrs { promote: true } } peers { permanent_uuid: "7785d6b6f5bf4dd5a2667857b24dcf8a" member_type: NON_VOTER last_known_addr { host: "127.29.234.60" port: 45643 } attrs { promote: true } } }
I20260924 16:12:58.368028 5095 sys_catalog.cc:458] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304 [sys.catalog]: This master's current role is: LEARNER
I20260924 16:12:58.376330 5204 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3639f056333442f859ccc6c600711ac [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d3639f056333442f859ccc6c600711ac" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d3639f056333442f859ccc6c600711ac" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 40099 } } peers { permanent_uuid: "d44fd4dbe8224ecf93501b1d97a76304" member_type: NON_VOTER last_known_addr { host: "127.29.234.61" port: 42395 } attrs { promote: true } } peers { permanent_uuid: "7785d6b6f5bf4dd5a2667857b24dcf8a" member_type: NON_VOTER last_known_addr { host: "127.29.234.60" port: 45643 } attrs { promote: true } } }
I20260924 16:12:58.376756 5204 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3639f056333442f859ccc6c600711ac [sys.catalog]: This master's current role is: LEADER
I20260924 16:12:58.375523 5203 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d3639f056333442f859ccc6c600711ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "7785d6b6f5bf4dd5a2667857b24dcf8a" member_type: NON_VOTER last_known_addr { host: "127.29.234.60" port: 45643 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.002s
I20260924 16:12:58.384174 5011 fs_manager.cc:709] Time spent opening directory manager: real 0.020s user 0.005s sys 0.000s
I20260924 16:12:58.385859 5014 raft_consensus.cc:1050] T 00000000000000000000000000000000 P d3639f056333442f859ccc6c600711ac: attempting to promote NON_VOTER d44fd4dbe8224ecf93501b1d97a76304 to VOTER
I20260924 16:12:58.388496 5014 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3639f056333442f859ccc6c600711ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d3639f056333442f859ccc6c600711ac" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 40099 } } peers { permanent_uuid: "d44fd4dbe8224ecf93501b1d97a76304" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 42395 } attrs { promote: false } } peers { permanent_uuid: "7785d6b6f5bf4dd5a2667857b24dcf8a" member_type: NON_VOTER last_known_addr { host: "127.29.234.60" port: 45643 } attrs { promote: true } }
I20260924 16:12:58.392278 5010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:58.391550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.60:45643 (local address 127.0.0.1:35482)
0924 16:12:58.391863 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:58.392033 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.60:45643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
I20260924 16:12:58.397681 5210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:58.398146 5011 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:58.398420 5011 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.006s sys 0.007s
I20260924 16:12:58.398520 5011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/master-2/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/master-2/wal
uuid: "7785d6b6f5bf4dd5a2667857b24dcf8a"
format_stamp: "Formatted at 2026-09-24 16:12:58 on dist-test-slave-v5j9"
server_key: "9ae97c93f3b3079faa734ab973547532"
server_key_iv: "4992c6a783253935974a68f5ec6d2e19"
server_key_version: "encryptionkey@0"
I20260924 16:12:58.399305 5010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:58.398839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.60:45643 (local address 127.0.0.1:35494)
0924 16:12:58.398999 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:58.399086 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.60:45643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 16:12:58.399307 5067 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304 [term 1 LEARNER]: Refusing update from remote peer d3639f056333442f859ccc6c600711ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20260924 16:12:58.399583 4434 consensus_peers.cc:487] T 00000000000000000000000000000000 P d3639f056333442f859ccc6c600711ac -> Peer 7785d6b6f5bf4dd5a2667857b24dcf8a (127.29.234.60:45643): Couldn't send request to peer 7785d6b6f5bf4dd5a2667857b24dcf8a. Status: Network error: Client connection negotiation failed: client connection to 127.29.234.60:45643: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 16:12:58.399983 5204 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d3639f056333442f859ccc6c600711ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "d44fd4dbe8224ecf93501b1d97a76304" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 42395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260924 16:12:58.402974 5010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:58.402663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.60:45643 (local address 127.0.0.1:35502)
0924 16:12:58.402795 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:58.402865 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.60:45643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 16:12:58.403744 5011 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 7785d6b6f5bf4dd5a2667857b24dcf8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:12:58.404076 5010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:58.403628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.60:45643 (local address 127.0.0.1:35510)
0924 16:12:58.403786 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:58.403863 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.60:45643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260924 16:12:58.404608 5204 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d3639f056333442f859ccc6c600711ac [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d44fd4dbe8224ecf93501b1d97a76304 (127.29.234.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d3639f056333442f859ccc6c600711ac" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 40099 } } peers { permanent_uuid: "d44fd4dbe8224ecf93501b1d97a76304" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 42395 } attrs { promote: false } } peers { permanent_uuid: "7785d6b6f5bf4dd5a2667857b24dcf8a" member_type: NON_VOTER last_known_addr { host: "127.29.234.60" port: 45643 } attrs { promote: true } } }
I20260924 16:12:58.404712 5067 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d44fd4dbe8224ecf93501b1d97a76304 (127.29.234.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d3639f056333442f859ccc6c600711ac" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 40099 } } peers { permanent_uuid: "d44fd4dbe8224ecf93501b1d97a76304" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 42395 } attrs { promote: false } } peers { permanent_uuid: "7785d6b6f5bf4dd5a2667857b24dcf8a" member_type: NON_VOTER last_known_addr { host: "127.29.234.60" port: 45643 } attrs { promote: true } } }
I20260924 16:12:58.409332 5095 sys_catalog.cc:455] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d3639f056333442f859ccc6c600711ac" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d3639f056333442f859ccc6c600711ac" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 40099 } } peers { permanent_uuid: "d44fd4dbe8224ecf93501b1d97a76304" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 42395 } attrs { promote: false } } peers { permanent_uuid: "7785d6b6f5bf4dd5a2667857b24dcf8a" member_type: NON_VOTER last_known_addr { host: "127.29.234.60" port: 45643 } attrs { promote: true } } }
I20260924 16:12:58.409423 5014 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3639f056333442f859ccc6c600711ac [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d3639f056333442f859ccc6c600711ac" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d3639f056333442f859ccc6c600711ac" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 40099 } } peers { permanent_uuid: "d44fd4dbe8224ecf93501b1d97a76304" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 42395 } attrs { promote: false } } peers { permanent_uuid: "7785d6b6f5bf4dd5a2667857b24dcf8a" member_type: NON_VOTER last_known_addr { host: "127.29.234.60" port: 45643 } attrs { promote: true } } }
I20260924 16:12:58.409708 5095 sys_catalog.cc:458] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:12:58.409811 5014 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3639f056333442f859ccc6c600711ac [sys.catalog]: This master's current role is: LEADER
I20260924 16:12:58.410183 5014 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3639f056333442f859ccc6c600711ac [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d3639f056333442f859ccc6c600711ac" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d3639f056333442f859ccc6c600711ac" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 40099 } } peers { permanent_uuid: "d44fd4dbe8224ecf93501b1d97a76304" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 42395 } attrs { promote: false } } peers { permanent_uuid: "7785d6b6f5bf4dd5a2667857b24dcf8a" member_type: NON_VOTER last_known_addr { host: "127.29.234.60" port: 45643 } attrs { promote: true } } }
I20260924 16:12:58.410458 5014 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3639f056333442f859ccc6c600711ac [sys.catalog]: This master's current role is: LEADER
I20260924 16:12:58.418973 5011 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 7785d6b6f5bf4dd5a2667857b24dcf8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 16:12:58.419174 5011 log.cc:1196] T 00000000000000000000000000000000 P 7785d6b6f5bf4dd5a2667857b24dcf8a: Deleting WAL directory at /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260924 16:12:58.419603 5011 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 7785d6b6f5bf4dd5a2667857b24dcf8a: Deleting consensus metadata
I20260924 16:12:58.419848 5011 master_runner.cc:315] Copying system tablet from 127.29.234.62:40099
I20260924 16:12:58.422396 5011 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 7785d6b6f5bf4dd5a2667857b24dcf8a: tablet copy: Beginning tablet copy session from remote peer at address 127.29.234.62:40099
I20260924 16:12:58.435041 4986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:58.423453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37036 (local address 127.29.234.62:40099)
0924 16:12:58.423683 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:58.423691 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:58.423797 (+ 106us) server_negotiation.cc:407] Connection header received
0924 16:12:58.424051 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:58.424057 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:58.424133 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:58.424238 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:58.425017 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:58.425834 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:58.427095 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:58.427513 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:58.427659 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:58.429611 (+ 1952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:58.429617 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:58.429621 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:58.429661 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:58.432294 (+ 2633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:58.432939 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:58.432948 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:58.432953 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:58.433100 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:58.433574 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:58.433579 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:58.433582 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:58.433866 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:58.434034 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:58.434423 (+ 389us) server_negotiation.cc:299] Negotiation successful
0924 16:12:58.434639 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260924 16:12:58.435091 5218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:58.423136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:40099 (local address 127.0.0.1:37036)
0924 16:12:58.423683 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:58.423703 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:58.423909 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:58.424262 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:58.424266 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:58.424677 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:58.424875 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:58.424884 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:58.426002 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:58.426008 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:58.426946 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:58.426958 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:58.427668 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:58.427674 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:58.427873 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:58.428563 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:58.428580 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:58.429418 (+ 838us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:58.432486 (+ 3068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:58.432490 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:58.432493 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:58.432819 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:58.433215 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:58.433220 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:58.433225 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:58.433456 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:58.433985 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:58.433992 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:58.434173 (+ 181us) client_negotiation.cc:769] Sending connection context
0924 16:12:58.434330 (+ 157us) client_negotiation.cc:240] Negotiation successful
0924 16:12:58.434645 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":569,"threads_started":1}
I20260924 16:12:58.435876 4467 tablet_copy_service.cc:140] P d3639f056333442f859ccc6c600711ac: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 7785d6b6f5bf4dd5a2667857b24dcf8a ({username='oryx', principal='oryx/127.29.234.60@KRBTEST.COM'} at 127.0.0.1:37036)
I20260924 16:12:58.436041 4467 tablet_copy_service.cc:161] P d3639f056333442f859ccc6c600711ac: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 7785d6b6f5bf4dd5a2667857b24dcf8a at {username='oryx', principal='oryx/127.29.234.60@KRBTEST.COM'} at 127.0.0.1:37036: session id = 7785d6b6f5bf4dd5a2667857b24dcf8a-00000000000000000000000000000000
I20260924 16:12:58.438571 4467 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P d3639f056333442f859ccc6c600711ac: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 16:12:58.440181 5011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 16:12:58.446125 5011 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 7785d6b6f5bf4dd5a2667857b24dcf8a: tablet copy: Starting download of 0 data blocks...
I20260924 16:12:58.446440 5011 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 7785d6b6f5bf4dd5a2667857b24dcf8a: tablet copy: Starting download of 1 WAL segments...
I20260924 16:12:58.448782 5011 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 7785d6b6f5bf4dd5a2667857b24dcf8a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 16:12:58.451985 4467 tablet_copy_service.cc:342] P d3639f056333442f859ccc6c600711ac: Request end of tablet copy session 7785d6b6f5bf4dd5a2667857b24dcf8a-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.29.234.60@KRBTEST.COM'} at 127.0.0.1:37036
I20260924 16:12:58.452119 4467 tablet_copy_service.cc:433] P d3639f056333442f859ccc6c600711ac: ending tablet copy session 7785d6b6f5bf4dd5a2667857b24dcf8a-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 7785d6b6f5bf4dd5a2667857b24dcf8a
I20260924 16:12:58.462572 5011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:58.465256 5220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:58.466048 5221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:58.478286 5223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:58.478667 5011 server_base.cc:1034] running on GCE node
I20260924 16:12:58.479022 5011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:58.479713 5011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:58.480976 5011 hybrid_clock.cc:648] HybridClock initialized: now 1790266378480933 us; error 37 us; skew 500 ppm
I20260924 16:12:58.481237 5011 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:12:58 dist-test-slave-v5j9 krb5kdc[4399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266378, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.234.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:58.483445 5011 init.cc:377] Logged in from keytab as oryx/127.29.234.60@KRBTEST.COM (short username oryx)
I20260924 16:12:58.484750 5011 webserver.cc:460] Webserver started at http://127.29.234.60:42689/ using document root <none> and password file <none>
I20260924 16:12:58.485067 5011 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:58.485178 5011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:58.487576 5011 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260924 16:12:58.489393 5230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:58.489881 5011 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260924 16:12:58.490028 5011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/master-2/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/master-2/wal
uuid: "7785d6b6f5bf4dd5a2667857b24dcf8a"
format_stamp: "Formatted at 2026-09-24 16:12:58 on dist-test-slave-v5j9"
server_key: "9ae97c93f3b3079faa734ab973547532"
server_key_iv: "4992c6a783253935974a68f5ec6d2e19"
server_key_version: "encryptionkey@0"
I20260924 16:12:58.490223 5011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:58.498260 5011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:58.501703 5011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:58.526763 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 4508
I20260924 16:12:58.536911 5011 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.60:45643
I20260924 16:12:58.536958 5282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.60:45643 every 8 connection(s)
I20260924 16:12:58.538100 5011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/master-2/data/info.pb
I20260924 16:12:58.543372 5283 sys_catalog.cc:263] Verifying existing consensus state
I20260924 16:12:58.547453 5283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7785d6b6f5bf4dd5a2667857b24dcf8a: Bootstrap starting.
I20260924 16:12:58.556392 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.1:40063
--local_ip_for_outbound_sockets=127.29.234.1
--tserver_master_addrs=127.29.234.62:40099
--webserver_port=32803
--webserver_interface=127.29.234.1
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/krb5kdc/oryx_127.29.234.1.keytab
--principal=oryx/127.29.234.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:40243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.29.234.62:40099,127.29.234.61:42395
--tserver_master_addrs=127.29.234.62:40099,127.29.234.61:42395,127.29.234.60:45643 with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:12:58.567636 5283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7785d6b6f5bf4dd5a2667857b24dcf8a: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 16:12:58.568246 5283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7785d6b6f5bf4dd5a2667857b24dcf8a: Bootstrap complete.
I20260924 16:12:58.569456 5283 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7785d6b6f5bf4dd5a2667857b24dcf8a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d3639f056333442f859ccc6c600711ac" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 40099 } } peers { permanent_uuid: "d44fd4dbe8224ecf93501b1d97a76304" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 42395 } attrs { promote: false } } peers { permanent_uuid: "7785d6b6f5bf4dd5a2667857b24dcf8a" member_type: NON_VOTER last_known_addr { host: "127.29.234.60" port: 45643 } attrs { promote: true } }
I20260924 16:12:58.569753 5283 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7785d6b6f5bf4dd5a2667857b24dcf8a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7785d6b6f5bf4dd5a2667857b24dcf8a, State: Initialized, Role: LEARNER
I20260924 16:12:58.570037 5283 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7785d6b6f5bf4dd5a2667857b24dcf8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d3639f056333442f859ccc6c600711ac" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 40099 } } peers { permanent_uuid: "d44fd4dbe8224ecf93501b1d97a76304" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 42395 } attrs { promote: false } } peers { permanent_uuid: "7785d6b6f5bf4dd5a2667857b24dcf8a" member_type: NON_VOTER last_known_addr { host: "127.29.234.60" port: 45643 } attrs { promote: true } }
I20260924 16:12:58.570883 5287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7785d6b6f5bf4dd5a2667857b24dcf8a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d3639f056333442f859ccc6c600711ac" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 40099 } } peers { permanent_uuid: "d44fd4dbe8224ecf93501b1d97a76304" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 42395 } attrs { promote: false } } peers { permanent_uuid: "7785d6b6f5bf4dd5a2667857b24dcf8a" member_type: NON_VOTER last_known_addr { host: "127.29.234.60" port: 45643 } attrs { promote: true } } }
I20260924 16:12:58.571164 5283 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7785d6b6f5bf4dd5a2667857b24dcf8a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:12:58.571365 5287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7785d6b6f5bf4dd5a2667857b24dcf8a [sys.catalog]: This master's current role is: LEARNER
I20260924 16:12:58.585695 5298 catalog_manager.cc:1259] Loaded cluster ID: e95643b1533745a68073f27a96aa4b1a
I20260924 16:12:58.585834 5298 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7785d6b6f5bf4dd5a2667857b24dcf8a: loading cluster ID for follower catalog manager: success
I20260924 16:12:58.589666 5298 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7785d6b6f5bf4dd5a2667857b24dcf8a: acquiring CA information for follower catalog manager: success
I20260924 16:12:58.592566 5298 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7785d6b6f5bf4dd5a2667857b24dcf8a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 16:12:58.891286 5010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:58.884367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.60:45643 (local address 127.0.0.1:35522)
0924 16:12:58.884562 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:58.884585 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:58.884744 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:58.885708 (+ 964us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:58.885715 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:58.885756 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:12:58.885970 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:58.885983 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:58.887254 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:58.887259 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:58.889179 (+ 1920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:58.889195 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:58.890480 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:58.890486 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:58.890707 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:58.890747 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 16:12:58.890856 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 16:12:58.891020 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260924 16:12:58.891294 5299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:58.884541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35522 (local address 127.29.234.60:45643)
0924 16:12:58.885237 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:58.885269 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:58.885292 (+ 23us) server_negotiation.cc:407] Connection header received
0924 16:12:58.885407 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:58.885412 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:58.885509 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:58.885639 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:12:58.886150 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:58.887099 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:58.889327 (+ 2228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:58.890291 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:58.890450 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:58.890677 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:58.890876 (+ 199us) server_negotiation.cc:299] Negotiation successful
0924 16:12:58.890969 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":437,"threads_started":1}
I20260924 16:12:58.892324 5257 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 7785d6b6f5bf4dd5a2667857b24dcf8a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260924 16:12:58.892953 5287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7785d6b6f5bf4dd5a2667857b24dcf8a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3639f056333442f859ccc6c600711ac. Latest consensus state: current_term: 1 leader_uuid: "d3639f056333442f859ccc6c600711ac" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d3639f056333442f859ccc6c600711ac" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 40099 } } peers { permanent_uuid: "d44fd4dbe8224ecf93501b1d97a76304" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 42395 } attrs { promote: false } } peers { permanent_uuid: "7785d6b6f5bf4dd5a2667857b24dcf8a" member_type: NON_VOTER last_known_addr { host: "127.29.234.60" port: 45643 } attrs { promote: true } } }
I20260924 16:12:58.893261 5287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7785d6b6f5bf4dd5a2667857b24dcf8a [sys.catalog]: This master's current role is: LEARNER
W20260924 16:12:58.912403 5286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:58.912731 5286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:58.912909 5286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:58.920482 5286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:58.920578 5286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:58.920691 5286 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.1
I20260924 16:12:58.930675 5286 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:40243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/krb5kdc/oryx_127.29.234.1.keytab
--principal=oryx/127.29.234.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.1:40063
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.234.1
--webserver_port=32803
--tserver_master_addrs=127.29.234.62:40099,127.29.234.61:42395,127.29.234.60:45643
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.1
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:12:58.931908 5286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:58.933287 5286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:58.947044 5304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:58.947124 5307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:58.947118 5305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:58.947742 5286 server_base.cc:1034] running on GCE node
I20260924 16:12:58.948324 5286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:58.949244 5286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:58.950452 5286 hybrid_clock.cc:648] HybridClock initialized: now 1790266378950383 us; error 62 us; skew 500 ppm
I20260924 16:12:58.950735 5286 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:12:58 dist-test-slave-v5j9 krb5kdc[4399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266378, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.234.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:58.954025 5286 init.cc:377] Logged in from keytab as oryx/127.29.234.1@KRBTEST.COM (short username oryx)
I20260924 16:12:58.955560 5286 webserver.cc:460] Webserver started at http://127.29.234.1:32803/ using document root <none> and password file <none>
I20260924 16:12:58.956256 5286 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:58.956404 5286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:58.962829 5286 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260924 16:12:58.965930 5314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:58.967363 5286 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260924 16:12:58.967551 5286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-0/wal
uuid: "76eabf034e914b95887d119fd3f8faa3"
format_stamp: "Formatted at 2026-09-24 16:12:55 on dist-test-slave-v5j9"
server_key: "e5f2615cff6d714a277cc84373f77b09"
server_key_iv: "11270d22460670904b1f8dd2c5adb700"
server_key_version: "encryptionkey@0"
I20260924 16:12:58.968516 5286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:58.999995 5286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:59.004139 5286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:59.004492 5286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:59.005538 5286 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:12:59.006923 5286 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:12:59.007009 5286 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:59.007088 5286 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:12:59.007148 5286 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:59.039857 5286 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.1:40063
I20260924 16:12:59.039949 5427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.1:40063 every 8 connection(s)
I20260924 16:12:59.041473 5286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
I20260924 16:12:59.051067 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 5286
I20260924 16:12:59.051458 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 4643
Sep 24 16:12:59 dist-test-slave-v5j9 krb5kdc[4399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266378, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.234.1@KRBTEST.COM for oryx/127.29.234.60@KRBTEST.COM
Sep 24 16:12:59 dist-test-slave-v5j9 krb5kdc[4399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266378, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.234.1@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM
Sep 24 16:12:59 dist-test-slave-v5j9 krb5kdc[4399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266378, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.234.1@KRBTEST.COM for oryx/127.29.234.61@KRBTEST.COM
Sep 24 16:12:59 dist-test-slave-v5j9 krb5kdc[4399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266374, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.29.234.2@KRBTEST.COM
I20260924 16:12:59.085268 5299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:59.045414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.1:43455 (local address 127.29.234.60:45643)
0924 16:12:59.045653 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:59.045663 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:59.059012 (+ 13349us) server_negotiation.cc:407] Connection header received
0924 16:12:59.064369 (+ 5357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:59.064378 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:59.064494 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:59.064616 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:59.068078 (+ 3462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:59.069220 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:59.072138 (+ 2918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:59.072809 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:59.072906 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:59.078098 (+ 5192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:59.078103 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:59.078107 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:59.078137 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:59.081468 (+ 3331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:59.082369 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:59.082375 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:59.082381 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:59.082521 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:59.083059 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:59.083066 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:59.083070 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:59.083485 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:59.083628 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:59.084672 (+ 1044us) server_negotiation.cc:299] Negotiation successful
0924 16:12:59.084923 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 16:12:59.086571 5432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:59.044154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.1:58365 (local address 127.29.234.62:40099)
0924 16:12:59.056511 (+ 12357us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:59.056521 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:59.066446 (+ 9925us) server_negotiation.cc:407] Connection header received
0924 16:12:59.066523 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:59.066527 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:59.066607 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:59.066727 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:59.068125 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:59.069277 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:59.072044 (+ 2767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:59.072706 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:59.072828 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:59.079327 (+ 6499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:59.079335 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:59.079340 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:59.079394 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:59.082864 (+ 3470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:59.083976 (+ 1112us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:59.083981 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:59.083984 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:59.084134 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:59.084580 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:59.084584 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:59.084587 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:59.084934 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:59.085076 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:59.085971 (+ 895us) server_negotiation.cc:299] Negotiation successful
0924 16:12:59.086227 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12179,"thread_start_us":12382,"threads_started":1}
I20260924 16:12:59.087792 5434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:59.045032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.60:45643 (local address 127.29.234.1:43455)
0924 16:12:59.058726 (+ 13694us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:59.058835 (+ 109us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:59.063973 (+ 5138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:59.065029 (+ 1056us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:59.065045 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:59.066270 (+ 1225us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:59.067841 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:59.067895 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:59.069441 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:59.069446 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:59.071855 (+ 2409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:59.071870 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:59.073247 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:59.073253 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:59.073661 (+ 408us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:59.074554 (+ 893us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:59.074576 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:59.077860 (+ 3284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:59.081629 (+ 3769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:59.081638 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:59.081657 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:59.082228 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:59.082669 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:59.082675 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:59.082679 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:59.082936 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:59.083730 (+ 794us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:59.083738 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:59.084206 (+ 468us) client_negotiation.cc:769] Sending connection context
0924 16:12:59.085594 (+ 1388us) client_negotiation.cc:240] Negotiation successful
0924 16:12:59.086227 (+ 633us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13194,"mutex_wait_us":248,"spinlock_wait_cycles":1280,"thread_start_us":16166,"threads_started":1}
I20260924 16:12:59.087766 5436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:59.045803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:40099 (local address 127.29.234.1:58365)
0924 16:12:59.065896 (+ 20093us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:59.065915 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:59.066070 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:59.066730 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:59.066736 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:59.067451 (+ 715us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:59.067916 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:59.067933 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:59.069516 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:59.069526 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:59.071866 (+ 2340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:59.071881 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:59.073144 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:59.073171 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:59.073668 (+ 497us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:59.074414 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:59.074458 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:59.079097 (+ 4639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:59.083013 (+ 3916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:59.083019 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:59.083026 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:59.083843 (+ 817us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:59.084266 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:59.084274 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:59.084278 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:59.084446 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:59.085135 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:59.085140 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:59.085284 (+ 144us) client_negotiation.cc:769] Sending connection context
0924 16:12:59.085786 (+ 502us) client_negotiation.cc:240] Negotiation successful
0924 16:12:59.087204 (+ 1418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19955,"mutex_wait_us":1340,"thread_start_us":20562,"threads_started":1}
I20260924 16:12:59.088425 5433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:59.044154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.1:53367 (local address 127.29.234.61:42395)
0924 16:12:59.056938 (+ 12784us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:59.056947 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:59.067674 (+ 10727us) server_negotiation.cc:407] Connection header received
0924 16:12:59.067743 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:59.067747 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:59.067841 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:59.067960 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:59.069137 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:59.070274 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:59.072319 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:59.073019 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:59.073125 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:59.080359 (+ 7234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:59.080365 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:59.080391 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:59.080435 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:59.084257 (+ 3822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:59.085055 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:59.085060 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:59.085064 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:59.085232 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:59.085788 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:59.085793 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:59.085796 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:59.086184 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:59.086321 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:59.087822 (+ 1501us) server_negotiation.cc:299] Negotiation successful
0924 16:12:59.088043 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12610,"thread_start_us":12799,"threads_started":1}
I20260924 16:12:59.088974 5435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:59.045032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.61:42395 (local address 127.29.234.1:53367)
0924 16:12:59.067402 (+ 22370us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:59.067421 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:59.067540 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:59.068169 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:59.068175 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:59.068697 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:59.068975 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:59.068986 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:59.070440 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:59.070444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:59.072172 (+ 1728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:59.072185 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:59.073434 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:59.073441 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:59.073689 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:59.074693 (+ 1004us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:59.074713 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:59.080145 (+ 5432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:59.084430 (+ 4285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:59.084435 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:59.084440 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:59.084923 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:59.085394 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:59.085402 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:59.085405 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:59.085609 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:59.086384 (+ 775us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:59.086389 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:59.086551 (+ 162us) client_negotiation.cc:769] Sending connection context
0924 16:12:59.086714 (+ 163us) client_negotiation.cc:240] Negotiation successful
0924 16:12:59.087023 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22172,"mutex_wait_us":1111,"spinlock_wait_cycles":28544,"thread_start_us":19535,"threads_started":1}
I20260924 16:12:59.089119 5438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:59.053139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55234 (local address 127.29.234.2:39763)
0924 16:12:59.064907 (+ 11768us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:59.064918 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:59.065724 (+ 806us) server_negotiation.cc:407] Connection header received
0924 16:12:59.065815 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:59.065824 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:59.066086 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:59.066287 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:59.068661 (+ 2374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:59.070159 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:59.072044 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:59.072996 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:59.073109 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:59.080276 (+ 7167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:59.080292 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:59.080311 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:59.080353 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:59.084863 (+ 4510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:59.085880 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:59.085891 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:59.085894 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:59.086048 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:59.086546 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:59.086551 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:59.086554 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:59.086969 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:59.087283 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:59.088431 (+ 1148us) server_negotiation.cc:299] Negotiation successful
0924 16:12:59.088710 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11578,"thread_start_us":11399,"threads_started":1}
I20260924 16:12:59.089989 5428 heartbeater.cc:347] Connected to a master server at 127.29.234.60:45643
I20260924 16:12:59.090447 5429 heartbeater.cc:347] Connected to a master server at 127.29.234.62:40099
I20260924 16:12:59.090382 5430 heartbeater.cc:347] Connected to a master server at 127.29.234.61:42395
I20260924 16:12:59.090595 5429 heartbeater.cc:464] Registering TS with master...
I20260924 16:12:59.090628 4770 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:12:59.090649 5430 heartbeater.cc:464] Registering TS with master...
I20260924 16:12:59.090378 5428 heartbeater.cc:464] Registering TS with master...
I20260924 16:12:59.091828 5430 heartbeater.cc:511] Master 127.29.234.61:42395 requested a full tablet report, sending...
I20260924 16:12:59.092736 5428 heartbeater.cc:511] Master 127.29.234.60:45643 requested a full tablet report, sending...
I20260924 16:12:59.094272 5057 ts_manager.cc:194] Registered new tserver with Master: 76eabf034e914b95887d119fd3f8faa3 (127.29.234.1:40063)
I20260924 16:12:59.094830 5247 ts_manager.cc:194] Registered new tserver with Master: 76eabf034e914b95887d119fd3f8faa3 (127.29.234.1:40063)
I20260924 16:12:59.094900 5429 heartbeater.cc:511] Master 127.29.234.62:40099 requested a full tablet report, sending...
I20260924 16:12:59.096074 4447 ts_manager.cc:194] Re-registered known tserver with Master: 76eabf034e914b95887d119fd3f8faa3 (127.29.234.1:40063)
I20260924 16:12:59.096889 4447 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.234.1@KRBTEST.COM'} at 127.29.234.1:58365
I20260924 16:12:59.178323 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 4643
I20260924 16:12:59.196133 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.2:39763
--local_ip_for_outbound_sockets=127.29.234.2
--tserver_master_addrs=127.29.234.62:40099
--webserver_port=42535
--webserver_interface=127.29.234.2
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/krb5kdc/oryx_127.29.234.2.keytab
--principal=oryx/127.29.234.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:40243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.29.234.62:40099,127.29.234.61:42395
--tserver_master_addrs=127.29.234.62:40099,127.29.234.61:42395,127.29.234.60:45643 with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:12:59.407469 5204 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3639f056333442f859ccc6c600711ac [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d3639f056333442f859ccc6c600711ac" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d3639f056333442f859ccc6c600711ac" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 40099 } } peers { permanent_uuid: "d44fd4dbe8224ecf93501b1d97a76304" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 42395 } attrs { promote: false } } peers { permanent_uuid: "7785d6b6f5bf4dd5a2667857b24dcf8a" member_type: NON_VOTER last_known_addr { host: "127.29.234.60" port: 45643 } attrs { promote: true } } }
I20260924 16:12:59.407950 5204 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3639f056333442f859ccc6c600711ac [sys.catalog]: This master's current role is: LEADER
I20260924 16:12:59.409761 5204 raft_consensus.cc:1050] T 00000000000000000000000000000000 P d3639f056333442f859ccc6c600711ac: attempting to promote NON_VOTER 7785d6b6f5bf4dd5a2667857b24dcf8a to VOTER
I20260924 16:12:59.410306 5204 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3639f056333442f859ccc6c600711ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d3639f056333442f859ccc6c600711ac" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 40099 } } peers { permanent_uuid: "d44fd4dbe8224ecf93501b1d97a76304" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 42395 } attrs { promote: false } } peers { permanent_uuid: "7785d6b6f5bf4dd5a2667857b24dcf8a" member_type: VOTER last_known_addr { host: "127.29.234.60" port: 45643 } attrs { promote: false } }
I20260924 16:12:59.412422 5257 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7785d6b6f5bf4dd5a2667857b24dcf8a [term 1 LEARNER]: Refusing update from remote peer d3639f056333442f859ccc6c600711ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260924 16:12:59.412549 5067 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304 [term 1 FOLLOWER]: Refusing update from remote peer d3639f056333442f859ccc6c600711ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260924 16:12:59.413110 5443 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d3639f056333442f859ccc6c600711ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "7785d6b6f5bf4dd5a2667857b24dcf8a" member_type: VOTER last_known_addr { host: "127.29.234.60" port: 45643 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260924 16:12:59.413520 5204 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d3639f056333442f859ccc6c600711ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "d44fd4dbe8224ecf93501b1d97a76304" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 42395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260924 16:12:59.416474 5443 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d3639f056333442f859ccc6c600711ac [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 7785d6b6f5bf4dd5a2667857b24dcf8a (127.29.234.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d3639f056333442f859ccc6c600711ac" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 40099 } } peers { permanent_uuid: "d44fd4dbe8224ecf93501b1d97a76304" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 42395 } attrs { promote: false } } peers { permanent_uuid: "7785d6b6f5bf4dd5a2667857b24dcf8a" member_type: VOTER last_known_addr { host: "127.29.234.60" port: 45643 } attrs { promote: false } } }
I20260924 16:12:59.417164 5067 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 7785d6b6f5bf4dd5a2667857b24dcf8a (127.29.234.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d3639f056333442f859ccc6c600711ac" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 40099 } } peers { permanent_uuid: "d44fd4dbe8224ecf93501b1d97a76304" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 42395 } attrs { promote: false } } peers { permanent_uuid: "7785d6b6f5bf4dd5a2667857b24dcf8a" member_type: VOTER last_known_addr { host: "127.29.234.60" port: 45643 } attrs { promote: false } } }
I20260924 16:12:59.418267 5257 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7785d6b6f5bf4dd5a2667857b24dcf8a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 7785d6b6f5bf4dd5a2667857b24dcf8a (127.29.234.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d3639f056333442f859ccc6c600711ac" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 40099 } } peers { permanent_uuid: "d44fd4dbe8224ecf93501b1d97a76304" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 42395 } attrs { promote: false } } peers { permanent_uuid: "7785d6b6f5bf4dd5a2667857b24dcf8a" member_type: VOTER last_known_addr { host: "127.29.234.60" port: 45643 } attrs { promote: false } } }
I20260924 16:12:59.420415 5204 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3639f056333442f859ccc6c600711ac [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d3639f056333442f859ccc6c600711ac" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d3639f056333442f859ccc6c600711ac" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 40099 } } peers { permanent_uuid: "d44fd4dbe8224ecf93501b1d97a76304" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 42395 } attrs { promote: false } } peers { permanent_uuid: "7785d6b6f5bf4dd5a2667857b24dcf8a" member_type: VOTER last_known_addr { host: "127.29.234.60" port: 45643 } attrs { promote: false } } }
I20260924 16:12:59.420667 5204 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3639f056333442f859ccc6c600711ac [sys.catalog]: This master's current role is: LEADER
I20260924 16:12:59.420499 5443 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3639f056333442f859ccc6c600711ac [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d3639f056333442f859ccc6c600711ac" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d3639f056333442f859ccc6c600711ac" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 40099 } } peers { permanent_uuid: "d44fd4dbe8224ecf93501b1d97a76304" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 42395 } attrs { promote: false } } peers { permanent_uuid: "7785d6b6f5bf4dd5a2667857b24dcf8a" member_type: VOTER last_known_addr { host: "127.29.234.60" port: 45643 } attrs { promote: false } } }
I20260924 16:12:59.420819 5443 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3639f056333442f859ccc6c600711ac [sys.catalog]: This master's current role is: LEADER
I20260924 16:12:59.420921 5204 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3639f056333442f859ccc6c600711ac [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d3639f056333442f859ccc6c600711ac" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d3639f056333442f859ccc6c600711ac" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 40099 } } peers { permanent_uuid: "d44fd4dbe8224ecf93501b1d97a76304" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 42395 } attrs { promote: false } } peers { permanent_uuid: "7785d6b6f5bf4dd5a2667857b24dcf8a" member_type: VOTER last_known_addr { host: "127.29.234.60" port: 45643 } attrs { promote: false } } }
I20260924 16:12:59.421211 5204 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3639f056333442f859ccc6c600711ac [sys.catalog]: This master's current role is: LEADER
I20260924 16:12:59.422050 5448 sys_catalog.cc:455] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d3639f056333442f859ccc6c600711ac" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d3639f056333442f859ccc6c600711ac" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 40099 } } peers { permanent_uuid: "d44fd4dbe8224ecf93501b1d97a76304" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 42395 } attrs { promote: false } } peers { permanent_uuid: "7785d6b6f5bf4dd5a2667857b24dcf8a" member_type: VOTER last_known_addr { host: "127.29.234.60" port: 45643 } attrs { promote: false } } }
I20260924 16:12:59.422489 5448 sys_catalog.cc:458] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:12:59.424034 5449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7785d6b6f5bf4dd5a2667857b24dcf8a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d3639f056333442f859ccc6c600711ac" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d3639f056333442f859ccc6c600711ac" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 40099 } } peers { permanent_uuid: "d44fd4dbe8224ecf93501b1d97a76304" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 42395 } attrs { promote: false } } peers { permanent_uuid: "7785d6b6f5bf4dd5a2667857b24dcf8a" member_type: VOTER last_known_addr { host: "127.29.234.60" port: 45643 } attrs { promote: false } } }
I20260924 16:12:59.424381 5449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7785d6b6f5bf4dd5a2667857b24dcf8a [sys.catalog]: This master's current role is: FOLLOWER
W20260924 16:12:59.531708 5441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:12:59.531997 5441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:12:59.532094 5441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:12:59.539322 5441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:12:59.539425 5441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:12:59.539624 5441 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.2
I20260924 16:12:59.549291 5441 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:40243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/krb5kdc/oryx_127.29.234.2.keytab
--principal=oryx/127.29.234.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.2:39763
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.234.2
--webserver_port=42535
--tserver_master_addrs=127.29.234.62:40099,127.29.234.61:42395,127.29.234.60:45643
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.2
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:12:59.550633 5441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:12:59.552003 5441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:12:59.566711 5457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:59.566650 5454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:12:59.566787 5455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:12:59.567802 5441 server_base.cc:1034] running on GCE node
I20260924 16:12:59.568554 5441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:12:59.569684 5441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:12:59.570914 5441 hybrid_clock.cc:648] HybridClock initialized: now 1790266379570867 us; error 55 us; skew 500 ppm
I20260924 16:12:59.571343 5441 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:12:59 dist-test-slave-v5j9 krb5kdc[4399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266379, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.234.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:12:59.574828 5441 init.cc:377] Logged in from keytab as oryx/127.29.234.2@KRBTEST.COM (short username oryx)
I20260924 16:12:59.576403 5441 webserver.cc:460] Webserver started at http://127.29.234.2:42535/ using document root <none> and password file <none>
I20260924 16:12:59.577219 5441 fs_manager.cc:359] Metadata directory not provided
I20260924 16:12:59.577419 5441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:12:59.582726 5441 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 16:12:59.585282 5465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:59.586874 5441 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 16:12:59.587056 5441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-1/wal
uuid: "14d33685ba3849c78f336923bf49e2d4"
format_stamp: "Formatted at 2026-09-24 16:12:56 on dist-test-slave-v5j9"
server_key: "07f9e816ca47ba80f628e3bceaababd9"
server_key_iv: "d92eadb560ea8760d9a5d5fe8ee84c64"
server_key_version: "encryptionkey@0"
I20260924 16:12:59.588016 5441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:12:59.619875 5441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:12:59.624393 5441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:12:59.624764 5441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:12:59.625907 5441 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:12:59.627470 5441 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:12:59.627605 5441 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:59.627748 5441 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:12:59.627854 5441 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:12:59.665293 5441 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.2:39763
I20260924 16:12:59.665366 5579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.2:39763 every 8 connection(s)
I20260924 16:12:59.667163 5441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
I20260924 16:12:59.667976 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 5441
I20260924 16:12:59.668321 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 4780
Sep 24 16:12:59 dist-test-slave-v5j9 krb5kdc[4399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266379, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.234.2@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM
Sep 24 16:12:59 dist-test-slave-v5j9 krb5kdc[4399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266379, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.234.2@KRBTEST.COM for oryx/127.29.234.61@KRBTEST.COM
Sep 24 16:12:59 dist-test-slave-v5j9 krb5kdc[4399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266379, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.234.2@KRBTEST.COM for oryx/127.29.234.60@KRBTEST.COM
Sep 24 16:12:59 dist-test-slave-v5j9 krb5kdc[4399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266374, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.29.234.3@KRBTEST.COM
I20260924 16:12:59.719126 5587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:59.676050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.2:41519 (local address 127.29.234.61:42395)
0924 16:12:59.690570 (+ 14520us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:59.690576 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:59.695080 (+ 4504us) server_negotiation.cc:407] Connection header received
0924 16:12:59.695154 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:59.695159 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:59.695320 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:59.695439 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:59.697687 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:59.698565 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:59.701604 (+ 3039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:59.702302 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:59.702425 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:59.708998 (+ 6573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:59.709007 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:59.709013 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:59.709100 (+ 87us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:59.715293 (+ 6193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:59.716425 (+ 1132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:59.716431 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:59.716435 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:59.716590 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:59.717097 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:59.717103 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:59.717107 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:59.717531 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:59.717683 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:59.718482 (+ 799us) server_negotiation.cc:299] Negotiation successful
0924 16:12:59.718723 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14366,"thread_start_us":14240,"threads_started":1}
I20260924 16:12:59.719144 5585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:59.670737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.2:41877 (local address 127.29.234.62:40099)
0924 16:12:59.694267 (+ 23530us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:59.694274 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:59.694299 (+ 25us) server_negotiation.cc:407] Connection header received
0924 16:12:59.694383 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:59.694390 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:59.694502 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:59.694623 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:59.697738 (+ 3115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:59.699215 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:59.702856 (+ 3641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:59.703533 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:59.703665 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:59.708636 (+ 4971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:59.708644 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:59.708649 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:59.708734 (+ 85us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:59.714476 (+ 5742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:59.715773 (+ 1297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:59.715780 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:59.715783 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:59.716016 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:59.716602 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:59.716609 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:59.716612 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:59.717005 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:59.717163 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:59.718448 (+ 1285us) server_negotiation.cc:299] Negotiation successful
0924 16:12:59.718699 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23359,"thread_start_us":26106,"threads_started":1}
I20260924 16:12:59.720297 5590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:59.675523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.60:45643 (local address 127.29.234.2:39263)
0924 16:12:59.693938 (+ 18415us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:59.693965 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:59.694160 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:59.695816 (+ 1656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:59.695824 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:59.696785 (+ 961us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:59.699146 (+ 2361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:59.699159 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:59.700802 (+ 1643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:59.700809 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:59.702747 (+ 1938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:59.702760 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:59.703841 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:59.703848 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:59.704067 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:59.704647 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:59.704669 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:59.709166 (+ 4497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:59.715665 (+ 6499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:59.715671 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:59.715675 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:59.716277 (+ 602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:59.716744 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:59.716748 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:59.716752 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:59.716956 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:59.717740 (+ 784us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:59.717746 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:59.718002 (+ 256us) client_negotiation.cc:769] Sending connection context
0924 16:12:59.718181 (+ 179us) client_negotiation.cc:240] Negotiation successful
0924 16:12:59.718722 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18239,"mutex_wait_us":1937,"spinlock_wait_cycles":11008,"thread_start_us":18914,"threads_started":1}
I20260924 16:12:59.720777 5586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:59.671813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:40099 (local address 127.29.234.2:41877)
0924 16:12:59.680854 (+ 9041us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:59.680949 (+ 95us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:59.684789 (+ 3840us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:59.695055 (+ 10266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:59.695071 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:59.696176 (+ 1105us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:59.697448 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:59.697480 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:59.700829 (+ 3349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:59.700834 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:59.702614 (+ 1780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:59.702628 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:59.703963 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:59.703970 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:59.704156 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:59.704817 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:59.704838 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:59.708264 (+ 3426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:59.714665 (+ 6401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:59.714677 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:59.714697 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:59.715593 (+ 896us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:59.716175 (+ 582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:59.716180 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:59.716184 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:59.716448 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:59.717239 (+ 791us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:59.717248 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:59.717745 (+ 497us) client_negotiation.cc:769] Sending connection context
0924 16:12:59.718093 (+ 348us) client_negotiation.cc:240] Negotiation successful
0924 16:12:59.720022 (+ 1929us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8472,"mutex_wait_us":1558,"thread_start_us":13453,"threads_started":1}
I20260924 16:12:59.721629 5588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:59.676058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.2:39263 (local address 127.29.234.60:45643)
0924 16:12:59.695137 (+ 19079us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:59.695146 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:59.695172 (+ 26us) server_negotiation.cc:407] Connection header received
0924 16:12:59.695248 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:59.695269 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:59.695390 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:59.695531 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:59.699354 (+ 3823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:59.700602 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:59.702932 (+ 2330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:59.703631 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:59.703807 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:59.709606 (+ 5799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:59.709615 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:59.709621 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:59.709674 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:59.715506 (+ 5832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:59.716425 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:59.716431 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:59.716435 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:59.716590 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:59.717097 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:59.717103 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:59.717107 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:59.717504 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:59.717661 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:59.720900 (+ 3239us) server_negotiation.cc:299] Negotiation successful
0924 16:12:59.721158 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18918,"thread_start_us":18798,"threads_started":1}
I20260924 16:12:59.722235 5581 heartbeater.cc:347] Connected to a master server at 127.29.234.62:40099
I20260924 16:12:59.722543 5580 heartbeater.cc:347] Connected to a master server at 127.29.234.60:45643
I20260924 16:12:59.722640 5581 heartbeater.cc:464] Registering TS with master...
I20260924 16:12:59.722697 5580 heartbeater.cc:464] Registering TS with master...
I20260924 16:12:59.723749 5580 heartbeater.cc:511] Master 127.29.234.60:45643 requested a full tablet report, sending...
I20260924 16:12:59.724164 5589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:59.675678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.61:42395 (local address 127.29.234.2:41519)
0924 16:12:59.694816 (+ 19138us) negotiation.cc:107] Waiting for socket to connect
0924 16:12:59.694836 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 16:12:59.694957 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:12:59.695673 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:12:59.695678 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:12:59.696382 (+ 704us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:12:59.697448 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:59.697480 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:59.698731 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:59.698738 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:59.701398 (+ 2660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:12:59.701421 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:12:59.702787 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:12:59.702798 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:12:59.703113 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:59.704397 (+ 1284us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:12:59.704459 (+ 62us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:12:59.708713 (+ 4254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:12:59.715473 (+ 6760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:59.715478 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:59.715482 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:59.716269 (+ 787us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:59.716735 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:12:59.716740 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:12:59.716743 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:12:59.716953 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:12:59.717740 (+ 787us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:12:59.717746 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:12:59.718002 (+ 256us) client_negotiation.cc:769] Sending connection context
0924 16:12:59.718181 (+ 179us) client_negotiation.cc:240] Negotiation successful
0924 16:12:59.723551 (+ 5370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18988,"mutex_wait_us":3854,"spinlock_wait_cycles":61440,"thread_start_us":18111,"threads_started":1}
I20260924 16:12:59.725099 5582 heartbeater.cc:347] Connected to a master server at 127.29.234.61:42395
I20260924 16:12:59.725167 5581 heartbeater.cc:511] Master 127.29.234.62:40099 requested a full tablet report, sending...
I20260924 16:12:59.725250 5582 heartbeater.cc:464] Registering TS with master...
I20260924 16:12:59.725785 5582 heartbeater.cc:511] Master 127.29.234.61:42395 requested a full tablet report, sending...
I20260924 16:12:59.725920 5247 ts_manager.cc:194] Registered new tserver with Master: 14d33685ba3849c78f336923bf49e2d4 (127.29.234.2:39763)
I20260924 16:12:59.726400 4447 ts_manager.cc:194] Re-registered known tserver with Master: 14d33685ba3849c78f336923bf49e2d4 (127.29.234.2:39763)
I20260924 16:12:59.726943 5057 ts_manager.cc:194] Registered new tserver with Master: 14d33685ba3849c78f336923bf49e2d4 (127.29.234.2:39763)
I20260924 16:12:59.727439 4447 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.234.2@KRBTEST.COM'} at 127.29.234.2:41877
I20260924 16:12:59.727608 5584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:12:59.669530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47348 (local address 127.29.234.3:34375)
0924 16:12:59.697610 (+ 28080us) server_negotiation.cc:206] Beginning negotiation
0924 16:12:59.697624 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 16:12:59.700043 (+ 2419us) server_negotiation.cc:407] Connection header received
0924 16:12:59.700125 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:12:59.700132 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:12:59.700343 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:12:59.700519 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:12:59.702586 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:59.704962 (+ 2376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:59.708984 (+ 4022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:12:59.710856 (+ 1872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:12:59.711000 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:12:59.718766 (+ 7766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:12:59.718781 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:12:59.718801 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:12:59.718852 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:12:59.723148 (+ 4296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:59.723833 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:59.723848 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:59.723851 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:59.724029 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:12:59.724503 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:12:59.724509 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:12:59.724514 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:12:59.724979 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:12:59.725289 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0924 16:12:59.726909 (+ 1620us) server_negotiation.cc:299] Negotiation successful
0924 16:12:59.727190 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27835,"thread_start_us":30231,"threads_started":1}
I20260924 16:12:59.728884 4904 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:12:59.823504 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 4780
I20260924 16:12:59.841523 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.3:34375
--local_ip_for_outbound_sockets=127.29.234.3
--tserver_master_addrs=127.29.234.62:40099
--webserver_port=42061
--webserver_interface=127.29.234.3
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/krb5kdc/oryx_127.29.234.3.keytab
--principal=oryx/127.29.234.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:40243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.29.234.62:40099,127.29.234.61:42395
--tserver_master_addrs=127.29.234.62:40099,127.29.234.61:42395,127.29.234.60:45643 with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:13:00.099432 5429 heartbeater.cc:503] Master 127.29.234.62:40099 was elected leader, sending a full tablet report...
W20260924 16:13:00.182353 5593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:13:00.182677 5593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:13:00.182852 5593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:13:00.190019 5593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:13:00.190181 5593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:13:00.190402 5593 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.3
I20260924 16:13:00.200093 5593 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:40243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/krb5kdc/oryx_127.29.234.3.keytab
--principal=oryx/127.29.234.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.3:34375
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.234.3
--webserver_port=42061
--tserver_master_addrs=127.29.234.62:40099,127.29.234.61:42395,127.29.234.60:45643
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.3
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:13:00.201437 5593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:13:00.202736 5593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:13:00.215727 5598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:13:00.215703 5601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:13:00.215749 5599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:13:00.218710 5593 server_base.cc:1034] running on GCE node
I20260924 16:13:00.219319 5593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:13:00.220549 5593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:13:00.221778 5593 hybrid_clock.cc:648] HybridClock initialized: now 1790266380221705 us; error 64 us; skew 500 ppm
I20260924 16:13:00.222152 5593 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:13:00 dist-test-slave-v5j9 krb5kdc[4399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266380, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.234.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:13:00.225494 5593 init.cc:377] Logged in from keytab as oryx/127.29.234.3@KRBTEST.COM (short username oryx)
I20260924 16:13:00.227027 5593 webserver.cc:460] Webserver started at http://127.29.234.3:42061/ using document root <none> and password file <none>
I20260924 16:13:00.227756 5593 fs_manager.cc:359] Metadata directory not provided
I20260924 16:13:00.227912 5593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:13:00.233606 5593 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:13:00.236392 5608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:13:00.237862 5593 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:13:00.238047 5593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-2/wal
uuid: "20193887ad994ccb8f3e9a4e4cad037c"
format_stamp: "Formatted at 2026-09-24 16:12:56 on dist-test-slave-v5j9"
server_key: "f5cd7d7be3d52eafc0569b6d4d35503e"
server_key_iv: "059ff158896fd8cc9e700f9be33b84f6"
server_key_version: "encryptionkey@0"
I20260924 16:13:00.239081 5593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:13:00.257524 5593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:13:00.262311 5593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:13:00.262619 5593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:13:00.263821 5593 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:13:00.265506 5593 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:13:00.265619 5593 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:13:00.265730 5593 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:13:00.265806 5593 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:13:00.298957 5593 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.3:34375
I20260924 16:13:00.299031 5722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.3:34375 every 8 connection(s)
I20260924 16:13:00.300405 5593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
I20260924 16:13:00.304622 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 5593
Sep 24 16:13:00 dist-test-slave-v5j9 krb5kdc[4399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266380, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.234.3@KRBTEST.COM for oryx/127.29.234.60@KRBTEST.COM
Sep 24 16:13:00 dist-test-slave-v5j9 krb5kdc[4399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266380, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.234.3@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM
I20260924 16:13:00.350155 5729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:00.304062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.60:45643 (local address 127.29.234.3:53253)
0924 16:13:00.320614 (+ 16552us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:00.320642 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:00.323002 (+ 2360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:00.324044 (+ 1042us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:00.324061 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:00.325315 (+ 1254us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:13:00.326415 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:00.326447 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:00.328377 (+ 1930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:00.328385 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:00.331128 (+ 2743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:00.331151 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:00.332909 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:00.332917 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:00.333225 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:00.334165 (+ 940us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:13:00.334199 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:13:00.337742 (+ 3543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:13:00.342596 (+ 4854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:13:00.342611 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:13:00.342638 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:13:00.343557 (+ 919us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:13:00.344109 (+ 552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:13:00.344118 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:13:00.344124 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:13:00.344438 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:13:00.345449 (+ 1011us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:13:00.345461 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:13:00.346009 (+ 548us) client_negotiation.cc:769] Sending connection context
0924 16:13:00.346432 (+ 423us) client_negotiation.cc:240] Negotiation successful
0924 16:13:00.347131 (+ 699us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16376,"mutex_wait_us":30,"thread_start_us":17372,"threads_started":1}
I20260924 16:13:00.350767 5727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:00.303870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.3:53253 (local address 127.29.234.60:45643)
0924 16:13:00.320158 (+ 16288us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:00.320168 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:00.321791 (+ 1623us) server_negotiation.cc:407] Connection header received
0924 16:13:00.323159 (+ 1368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:00.323169 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:00.323386 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:00.323502 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:00.326676 (+ 3174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:00.328139 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:00.331772 (+ 3633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:00.332414 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:00.332510 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:00.338188 (+ 5678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:13:00.338197 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:13:00.338202 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:13:00.338248 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:13:00.342459 (+ 4211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:00.343755 (+ 1296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:00.343761 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:00.343767 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:00.343959 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:00.344679 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:00.344687 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:00.344693 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:00.345144 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:13:00.345326 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:00.348618 (+ 3292us) server_negotiation.cc:299] Negotiation successful
0924 16:13:00.348886 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16111,"thread_start_us":17791,"threads_started":1}
I20260924 16:13:00.354064 5723 heartbeater.cc:347] Connected to a master server at 127.29.234.60:45643
I20260924 16:13:00.354490 5723 heartbeater.cc:464] Registering TS with master...
I20260924 16:13:00.355504 5723 heartbeater.cc:511] Master 127.29.234.60:45643 requested a full tablet report, sending...
I20260924 16:13:00.357726 5247 ts_manager.cc:194] Registered new tserver with Master: 20193887ad994ccb8f3e9a4e4cad037c (127.29.234.3:34375)
I20260924 16:13:00.360740 5730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:00.304088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:40099 (local address 127.29.234.3:38773)
0924 16:13:00.321176 (+ 17088us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:00.321195 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:00.322916 (+ 1721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:00.334375 (+ 11459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:00.334384 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:00.335337 (+ 953us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:13:00.335838 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:00.335855 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:00.338218 (+ 2363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:00.338224 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:00.339800 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:00.339819 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:00.340996 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:00.341004 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:00.341198 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:00.341873 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:13:00.341899 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:13:00.345364 (+ 3465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:13:00.350482 (+ 5118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:13:00.350491 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:13:00.350497 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:13:00.351154 (+ 657us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:13:00.351665 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:13:00.351687 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:13:00.351693 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:13:00.351875 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:13:00.359339 (+ 7464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:13:00.359347 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:13:00.359550 (+ 203us) client_negotiation.cc:769] Sending connection context
0924 16:13:00.359725 (+ 175us) client_negotiation.cc:240] Negotiation successful
0924 16:13:00.360127 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16958,"mutex_wait_us":403,"thread_start_us":16862,"threads_started":1}
I20260924 16:13:00.362082 5731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:00.304802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.3:38773 (local address 127.29.234.62:40099)
0924 16:13:00.333976 (+ 29174us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:00.333986 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:00.334010 (+ 24us) server_negotiation.cc:407] Connection header received
0924 16:13:00.334077 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:00.334085 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:00.334171 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:00.334299 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:00.336922 (+ 2623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:00.338058 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:00.340016 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:00.340782 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:00.341568 (+ 786us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:00.345665 (+ 4097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:13:00.345687 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:13:00.345695 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:13:00.345752 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:13:00.350300 (+ 4548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:00.351347 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:00.351356 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:00.351362 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:00.351528 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:00.352036 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:00.352044 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:00.352049 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:00.359169 (+ 7120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:13:00.361174 (+ 2005us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:00.361337 (+ 163us) server_negotiation.cc:299] Negotiation successful
0924 16:13:00.361646 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29016,"thread_start_us":33094,"threads_started":1}
I20260924 16:13:00.362869 5724 heartbeater.cc:347] Connected to a master server at 127.29.234.62:40099
I20260924 16:13:00.363021 5724 heartbeater.cc:464] Registering TS with master...
I20260924 16:13:00.363533 5724 heartbeater.cc:511] Master 127.29.234.62:40099 requested a full tablet report, sending...
I20260924 16:13:00.364738 4447 ts_manager.cc:194] Re-registered known tserver with Master: 20193887ad994ccb8f3e9a4e4cad037c (127.29.234.3:34375)
I20260924 16:13:00.366182 4447 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.234.3@KRBTEST.COM'} at 127.29.234.3:38773
Sep 24 16:13:00 dist-test-slave-v5j9 krb5kdc[4399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266380, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.234.3@KRBTEST.COM for oryx/127.29.234.61@KRBTEST.COM
I20260924 16:13:00.383329 5737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:00.338246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37038 (local address 127.29.234.62:40099)
0924 16:13:00.363306 (+ 25060us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:00.363312 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:00.363636 (+ 324us) server_negotiation.cc:407] Connection header received
0924 16:13:00.363767 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:00.363774 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:00.363849 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:00.363956 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:00.367033 (+ 3077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:00.368270 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:00.370234 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:00.370847 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:00.370950 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:00.374977 (+ 4027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:13:00.374985 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:13:00.374993 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:13:00.375036 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:13:00.379124 (+ 4088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:00.380152 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:00.380158 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:00.380161 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:00.380331 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:00.380776 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:00.380781 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:00.380784 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:00.381145 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:13:00.381443 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:00.382627 (+ 1184us) server_negotiation.cc:299] Negotiation successful
0924 16:13:00.382886 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23863,"thread_start_us":23720,"threads_started":1}
I20260924 16:13:00.384029 5732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:00.305153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.3:44849 (local address 127.29.234.61:42395)
0924 16:13:00.332149 (+ 26996us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:00.332155 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:00.332176 (+ 21us) server_negotiation.cc:407] Connection header received
0924 16:13:00.332238 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:00.332243 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:00.332327 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:00.334505 (+ 2178us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:00.334690 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:00.335784 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:00.337843 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:00.366661 (+ 28818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:00.372457 (+ 5796us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:00.374211 (+ 1754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:13:00.374219 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:13:00.374225 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:13:00.374282 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:13:00.380538 (+ 6256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:00.381666 (+ 1128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:00.381684 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:00.381689 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:00.381875 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:00.382411 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:00.382419 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:00.382425 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:00.382770 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:13:00.382929 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:00.383366 (+ 437us) server_negotiation.cc:299] Negotiation successful
0924 16:13:00.383590 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26854,"thread_start_us":27274,"threads_started":1}
I20260924 16:13:00.384034 5728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:00.304062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.61:42395 (local address 127.29.234.3:44849)
0924 16:13:00.314937 (+ 10875us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:00.314988 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:00.322472 (+ 7484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:00.332620 (+ 10148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:00.332625 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:00.333163 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:13:00.333467 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:00.333479 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:00.336005 (+ 2526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:00.336012 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:00.337623 (+ 1611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:00.337640 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:00.366932 (+ 29292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:00.366943 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:00.369120 (+ 2177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:00.369838 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:13:00.369862 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:13:00.373447 (+ 3585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:13:00.380771 (+ 7324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:13:00.380779 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:13:00.380786 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:13:00.381470 (+ 684us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:13:00.382035 (+ 565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:13:00.382044 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:13:00.382050 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:13:00.382255 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:13:00.382925 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:13:00.382929 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:13:00.383070 (+ 141us) client_negotiation.cc:769] Sending connection context
0924 16:13:00.383269 (+ 199us) client_negotiation.cc:240] Negotiation successful
0924 16:13:00.383605 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10549,"thread_start_us":10333,"threads_started":1}
I20260924 16:13:00.385130 5725 heartbeater.cc:347] Connected to a master server at 127.29.234.61:42395
I20260924 16:13:00.385300 5725 heartbeater.cc:464] Registering TS with master...
I20260924 16:13:00.385840 5725 heartbeater.cc:511] Master 127.29.234.61:42395 requested a full tablet report, sending...
I20260924 16:13:00.387300 5057 ts_manager.cc:194] Registered new tserver with Master: 20193887ad994ccb8f3e9a4e4cad037c (127.29.234.3:34375)
I20260924 16:13:00.388885 4447 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37038:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 16:13:00.399291 5739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:00.339386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52122 (local address 127.29.234.61:42395)
0924 16:13:00.379676 (+ 40290us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:00.379682 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:00.380519 (+ 837us) server_negotiation.cc:407] Connection header received
0924 16:13:00.380582 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:00.380587 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:00.380664 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:00.380780 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:00.383594 (+ 2814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:00.386226 (+ 2632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:00.388193 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:00.388639 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:00.388791 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:00.391990 (+ 3199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:13:00.391997 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:13:00.392001 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:13:00.392049 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:13:00.395914 (+ 3865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:00.396808 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:00.396814 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:00.396818 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:00.396998 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:00.397532 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:00.397537 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:00.397541 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:00.397918 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:13:00.398185 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:00.398597 (+ 412us) server_negotiation.cc:299] Negotiation successful
0924 16:13:00.398831 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40113,"thread_start_us":41547,"threads_started":1}
W20260924 16:13:00.400935 4447 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:13:00.401772 5727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:00.348601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35538 (local address 127.29.234.60:45643)
0924 16:13:00.351486 (+ 2885us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:00.351493 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:00.384773 (+ 33280us) server_negotiation.cc:407] Connection header received
0924 16:13:00.385031 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:00.385036 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:00.385148 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:00.385241 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:00.387695 (+ 2454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:00.388762 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:00.390893 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:00.391576 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:00.391674 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:00.395357 (+ 3683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:13:00.395365 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:13:00.395370 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:13:00.395416 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:13:00.398669 (+ 3253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:00.399367 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:00.399373 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:00.399377 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:00.399540 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:00.399981 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:00.399986 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:00.399989 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:00.400392 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:13:00.400633 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:00.401126 (+ 493us) server_negotiation.cc:299] Negotiation successful
0924 16:13:00.401397 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2742,"thread_start_us":30510,"threads_started":1}
I20260924 16:13:00.449254 5752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:00.421560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55238 (local address 127.29.234.2:39763)
0924 16:13:00.437657 (+ 16097us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:00.437670 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:00.437721 (+ 51us) server_negotiation.cc:407] Connection header received
0924 16:13:00.437864 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:00.437877 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:00.438863 (+ 986us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:00.439121 (+ 258us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:13:00.439869 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:00.442285 (+ 2416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:00.445452 (+ 3167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:00.447684 (+ 2232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:00.447871 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:00.448157 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:00.448621 (+ 464us) server_negotiation.cc:299] Negotiation successful
0924 16:13:00.448869 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15871,"thread_start_us":16719,"threads_started":1}
I20260924 16:13:00.449275 5751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:00.421326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.2:39763 (local address 127.0.0.1:55238)
0924 16:13:00.436677 (+ 15351us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:00.436704 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:00.436888 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:00.439133 (+ 2245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:00.439140 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:00.439176 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:13:00.439641 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:00.439657 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:00.442456 (+ 2799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:00.442464 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:00.445179 (+ 2715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:00.445200 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:00.448084 (+ 2884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:00.448093 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:00.448406 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:00.448442 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 16:13:00.448534 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 16:13:00.448689 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15164,"thread_start_us":15633,"threads_started":1}
I20260924 16:13:00.454078 5513 tablet_service.cc:1467] Processing CreateTablet for tablet 4e6536dc57764ab18a71722feeff91ec (DEFAULT_TABLE table=test-table [id=e338f019f6f647fab6f690debfc0c753]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:13:00.457954 5513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e6536dc57764ab18a71722feeff91ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:13:00.459573 5754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:00.421715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:40063 (local address 127.0.0.1:50298)
0924 16:13:00.447971 (+ 26256us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:00.447996 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:00.448182 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:00.449597 (+ 1415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:00.449604 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:00.449645 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:13:00.450688 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:00.450703 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:00.452996 (+ 2293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:00.453001 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:00.456588 (+ 3587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:00.456606 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:00.458652 (+ 2046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:00.458662 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:00.458867 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:00.458910 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 16:13:00.459012 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 16:13:00.459164 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25988,"thread_start_us":27970,"threads_started":1}
I20260924 16:13:00.460045 5755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:00.422339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50298 (local address 127.29.234.1:40063)
0924 16:13:00.447025 (+ 24686us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:00.447033 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:00.448380 (+ 1347us) server_negotiation.cc:407] Connection header received
0924 16:13:00.448498 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:00.448510 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:00.448867 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:00.449118 (+ 251us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:13:00.450937 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:00.452828 (+ 1891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:00.456929 (+ 4101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:00.457978 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:00.458134 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:00.458429 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:00.459151 (+ 722us) server_negotiation.cc:299] Negotiation successful
0924 16:13:00.459404 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24532,"thread_start_us":24689,"threads_started":1}
I20260924 16:13:00.463815 5362 tablet_service.cc:1467] Processing CreateTablet for tablet 4e6536dc57764ab18a71722feeff91ec (DEFAULT_TABLE table=test-table [id=e338f019f6f647fab6f690debfc0c753]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:13:00.465420 5362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e6536dc57764ab18a71722feeff91ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:13:00.465680 5751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:00.432366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.3:34375 (local address 127.0.0.1:47356)
0924 16:13:00.449830 (+ 17464us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:00.449850 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:00.449993 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:00.457462 (+ 7469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:00.457466 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:00.457493 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:13:00.458013 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:00.458030 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:00.460555 (+ 2525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:00.460562 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:00.462413 (+ 1851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:00.462428 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:00.464568 (+ 2140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:00.464577 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:00.464803 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:00.464877 (+ 74us) client_negotiation.cc:769] Sending connection context
0924 16:13:00.465048 (+ 171us) client_negotiation.cc:240] Negotiation successful
0924 16:13:00.465191 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17375,"thread_start_us":24096,"threads_started":1}
I20260924 16:13:00.465780 5757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:00.433055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47356 (local address 127.29.234.3:34375)
0924 16:13:00.454413 (+ 21358us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:00.454425 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:00.454493 (+ 68us) server_negotiation.cc:407] Connection header received
0924 16:13:00.454625 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:00.454638 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:00.455049 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:00.455314 (+ 265us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:13:00.458410 (+ 3096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:00.460412 (+ 2002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:00.462576 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:00.464301 (+ 1725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:00.464440 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:00.464702 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:00.465126 (+ 424us) server_negotiation.cc:299] Negotiation successful
0924 16:13:00.465403 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21046,"thread_start_us":20616,"threads_started":1}
I20260924 16:13:00.470275 5657 tablet_service.cc:1467] Processing CreateTablet for tablet 4e6536dc57764ab18a71722feeff91ec (DEFAULT_TABLE table=test-table [id=e338f019f6f647fab6f690debfc0c753]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:13:00.471724 5657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e6536dc57764ab18a71722feeff91ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:13:00.475132 5758 tablet_bootstrap.cc:492] T 4e6536dc57764ab18a71722feeff91ec P 14d33685ba3849c78f336923bf49e2d4: Bootstrap starting.
I20260924 16:13:00.479278 5758 tablet_bootstrap.cc:654] T 4e6536dc57764ab18a71722feeff91ec P 14d33685ba3849c78f336923bf49e2d4: Neither blocks nor log segments found. Creating new log.
I20260924 16:13:00.481539 5758 log.cc:824] T 4e6536dc57764ab18a71722feeff91ec P 14d33685ba3849c78f336923bf49e2d4: Log is configured to *not* fsync() on all Append() calls
I20260924 16:13:00.485951 5758 tablet_bootstrap.cc:492] T 4e6536dc57764ab18a71722feeff91ec P 14d33685ba3849c78f336923bf49e2d4: No bootstrap required, opened a new log
I20260924 16:13:00.486325 5758 ts_tablet_manager.cc:1397] T 4e6536dc57764ab18a71722feeff91ec P 14d33685ba3849c78f336923bf49e2d4: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260924 16:13:00.490345 5760 tablet_bootstrap.cc:492] T 4e6536dc57764ab18a71722feeff91ec P 76eabf034e914b95887d119fd3f8faa3: Bootstrap starting.
I20260924 16:13:00.497964 5760 tablet_bootstrap.cc:654] T 4e6536dc57764ab18a71722feeff91ec P 76eabf034e914b95887d119fd3f8faa3: Neither blocks nor log segments found. Creating new log.
I20260924 16:13:00.498008 5758 raft_consensus.cc:348] T 4e6536dc57764ab18a71722feeff91ec P 14d33685ba3849c78f336923bf49e2d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d33685ba3849c78f336923bf49e2d4" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 39763 } } peers { permanent_uuid: "76eabf034e914b95887d119fd3f8faa3" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 40063 } } peers { permanent_uuid: "20193887ad994ccb8f3e9a4e4cad037c" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 34375 } }
I20260924 16:13:00.498365 5761 tablet_bootstrap.cc:492] T 4e6536dc57764ab18a71722feeff91ec P 20193887ad994ccb8f3e9a4e4cad037c: Bootstrap starting.
I20260924 16:13:00.498520 5758 raft_consensus.cc:374] T 4e6536dc57764ab18a71722feeff91ec P 14d33685ba3849c78f336923bf49e2d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:13:00.498625 5758 raft_consensus.cc:729] T 4e6536dc57764ab18a71722feeff91ec P 14d33685ba3849c78f336923bf49e2d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14d33685ba3849c78f336923bf49e2d4, State: Initialized, Role: FOLLOWER
I20260924 16:13:00.499521 5760 log.cc:824] T 4e6536dc57764ab18a71722feeff91ec P 76eabf034e914b95887d119fd3f8faa3: Log is configured to *not* fsync() on all Append() calls
I20260924 16:13:00.499809 5758 consensus_queue.cc:260] T 4e6536dc57764ab18a71722feeff91ec P 14d33685ba3849c78f336923bf49e2d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d33685ba3849c78f336923bf49e2d4" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 39763 } } peers { permanent_uuid: "76eabf034e914b95887d119fd3f8faa3" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 40063 } } peers { permanent_uuid: "20193887ad994ccb8f3e9a4e4cad037c" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 34375 } }
I20260924 16:13:00.503016 5761 tablet_bootstrap.cc:654] T 4e6536dc57764ab18a71722feeff91ec P 20193887ad994ccb8f3e9a4e4cad037c: Neither blocks nor log segments found. Creating new log.
I20260924 16:13:00.505782 5761 log.cc:824] T 4e6536dc57764ab18a71722feeff91ec P 20193887ad994ccb8f3e9a4e4cad037c: Log is configured to *not* fsync() on all Append() calls
I20260924 16:13:00.507161 5760 tablet_bootstrap.cc:492] T 4e6536dc57764ab18a71722feeff91ec P 76eabf034e914b95887d119fd3f8faa3: No bootstrap required, opened a new log
I20260924 16:13:00.507475 5760 ts_tablet_manager.cc:1397] T 4e6536dc57764ab18a71722feeff91ec P 76eabf034e914b95887d119fd3f8faa3: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.012s
I20260924 16:13:00.508566 5758 ts_tablet_manager.cc:1428] T 4e6536dc57764ab18a71722feeff91ec P 14d33685ba3849c78f336923bf49e2d4: Time spent starting tablet: real 0.022s user 0.010s sys 0.010s
I20260924 16:13:00.509737 5761 tablet_bootstrap.cc:492] T 4e6536dc57764ab18a71722feeff91ec P 20193887ad994ccb8f3e9a4e4cad037c: No bootstrap required, opened a new log
I20260924 16:13:00.510088 5761 ts_tablet_manager.cc:1397] T 4e6536dc57764ab18a71722feeff91ec P 20193887ad994ccb8f3e9a4e4cad037c: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260924 16:13:00.510838 5581 heartbeater.cc:503] Master 127.29.234.62:40099 was elected leader, sending a full tablet report...
I20260924 16:13:00.519996 5760 raft_consensus.cc:348] T 4e6536dc57764ab18a71722feeff91ec P 76eabf034e914b95887d119fd3f8faa3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d33685ba3849c78f336923bf49e2d4" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 39763 } } peers { permanent_uuid: "76eabf034e914b95887d119fd3f8faa3" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 40063 } } peers { permanent_uuid: "20193887ad994ccb8f3e9a4e4cad037c" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 34375 } }
I20260924 16:13:00.520527 5760 raft_consensus.cc:374] T 4e6536dc57764ab18a71722feeff91ec P 76eabf034e914b95887d119fd3f8faa3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:13:00.520653 5760 raft_consensus.cc:729] T 4e6536dc57764ab18a71722feeff91ec P 76eabf034e914b95887d119fd3f8faa3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76eabf034e914b95887d119fd3f8faa3, State: Initialized, Role: FOLLOWER
I20260924 16:13:00.521567 5760 consensus_queue.cc:260] T 4e6536dc57764ab18a71722feeff91ec P 76eabf034e914b95887d119fd3f8faa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d33685ba3849c78f336923bf49e2d4" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 39763 } } peers { permanent_uuid: "76eabf034e914b95887d119fd3f8faa3" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 40063 } } peers { permanent_uuid: "20193887ad994ccb8f3e9a4e4cad037c" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 34375 } }
I20260924 16:13:00.523697 5760 ts_tablet_manager.cc:1428] T 4e6536dc57764ab18a71722feeff91ec P 76eabf034e914b95887d119fd3f8faa3: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260924 16:13:00.524300 5761 raft_consensus.cc:348] T 4e6536dc57764ab18a71722feeff91ec P 20193887ad994ccb8f3e9a4e4cad037c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d33685ba3849c78f336923bf49e2d4" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 39763 } } peers { permanent_uuid: "76eabf034e914b95887d119fd3f8faa3" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 40063 } } peers { permanent_uuid: "20193887ad994ccb8f3e9a4e4cad037c" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 34375 } }
I20260924 16:13:00.524781 5761 raft_consensus.cc:374] T 4e6536dc57764ab18a71722feeff91ec P 20193887ad994ccb8f3e9a4e4cad037c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:13:00.524885 5761 raft_consensus.cc:729] T 4e6536dc57764ab18a71722feeff91ec P 20193887ad994ccb8f3e9a4e4cad037c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20193887ad994ccb8f3e9a4e4cad037c, State: Initialized, Role: FOLLOWER
I20260924 16:13:00.525599 5761 consensus_queue.cc:260] T 4e6536dc57764ab18a71722feeff91ec P 20193887ad994ccb8f3e9a4e4cad037c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d33685ba3849c78f336923bf49e2d4" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 39763 } } peers { permanent_uuid: "76eabf034e914b95887d119fd3f8faa3" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 40063 } } peers { permanent_uuid: "20193887ad994ccb8f3e9a4e4cad037c" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 34375 } }
I20260924 16:13:00.527573 5761 ts_tablet_manager.cc:1428] T 4e6536dc57764ab18a71722feeff91ec P 20193887ad994ccb8f3e9a4e4cad037c: Time spent starting tablet: real 0.017s user 0.006s sys 0.009s
I20260924 16:13:00.528872 5724 heartbeater.cc:503] Master 127.29.234.62:40099 was elected leader, sending a full tablet report...
W20260924 16:13:00.553900 5726 tablet.cc:2369] T 4e6536dc57764ab18a71722feeff91ec P 20193887ad994ccb8f3e9a4e4cad037c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 16:13:00.564505 5431 tablet.cc:2369] T 4e6536dc57764ab18a71722feeff91ec P 76eabf034e914b95887d119fd3f8faa3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 16:13:00.691128 5591 tablet.cc:2369] T 4e6536dc57764ab18a71722feeff91ec P 14d33685ba3849c78f336923bf49e2d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:13:00.832156 5763 raft_consensus.cc:482] T 4e6536dc57764ab18a71722feeff91ec P 14d33685ba3849c78f336923bf49e2d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:13:00.832559 5763 raft_consensus.cc:504] T 4e6536dc57764ab18a71722feeff91ec P 14d33685ba3849c78f336923bf49e2d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d33685ba3849c78f336923bf49e2d4" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 39763 } } peers { permanent_uuid: "76eabf034e914b95887d119fd3f8faa3" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 40063 } } peers { permanent_uuid: "20193887ad994ccb8f3e9a4e4cad037c" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 34375 } }
I20260924 16:13:00.834517 5763 leader_election.cc:288] T 4e6536dc57764ab18a71722feeff91ec P 14d33685ba3849c78f336923bf49e2d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76eabf034e914b95887d119fd3f8faa3 (127.29.234.1:40063), 20193887ad994ccb8f3e9a4e4cad037c (127.29.234.3:34375)
I20260924 16:13:00.844147 5755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:00.835138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.2:54599 (local address 127.29.234.1:40063)
0924 16:13:00.835443 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:00.835452 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:00.836471 (+ 1019us) server_negotiation.cc:407] Connection header received
0924 16:13:00.836786 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:00.836790 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:00.836862 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:00.836946 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:13:00.837811 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:00.838658 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:00.841635 (+ 2977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:00.842914 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:00.843093 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:00.843156 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:00.843725 (+ 569us) server_negotiation.cc:299] Negotiation successful
0924 16:13:00.843889 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260924 16:13:00.844153 5768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:00.834777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.3:34375 (local address 127.29.234.2:57373)
0924 16:13:00.836511 (+ 1734us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:00.836527 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:00.836625 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:00.836978 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:00.836983 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:00.837009 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:13:00.837541 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:00.837561 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:00.838991 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:00.839000 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:00.841440 (+ 2440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:00.841456 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:00.843141 (+ 1685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:00.843147 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:00.843323 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:00.843463 (+ 140us) client_negotiation.cc:769] Sending connection context
0924 16:13:00.843590 (+ 127us) client_negotiation.cc:240] Negotiation successful
0924 16:13:00.843737 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1631,"mutex_wait_us":86,"thread_start_us":1489,"threads_started":1}
I20260924 16:13:00.844489 5757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:00.835158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.2:57373 (local address 127.29.234.3:34375)
0924 16:13:00.835660 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:00.835669 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:00.836649 (+ 980us) server_negotiation.cc:407] Connection header received
0924 16:13:00.836743 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:00.836748 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:00.836836 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:00.836935 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:13:00.837733 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:00.838835 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:00.841623 (+ 2788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:00.842914 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:00.843082 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:00.843156 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:00.843619 (+ 463us) server_negotiation.cc:299] Negotiation successful
0924 16:13:00.843928 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328}
I20260924 16:13:00.844897 5767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:00.834683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:40063 (local address 127.29.234.2:54599)
0924 16:13:00.836318 (+ 1635us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:00.836357 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:00.836605 (+ 248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:00.837160 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:00.837165 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:00.837200 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:13:00.837650 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:00.837662 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:00.838808 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:00.838811 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:00.841440 (+ 2629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:00.841456 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:00.843341 (+ 1885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:00.843349 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:00.843524 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:00.843558 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 16:13:00.844381 (+ 823us) client_negotiation.cc:240] Negotiation successful
0924 16:13:00.844514 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1419,"mutex_wait_us":90,"thread_start_us":1190,"threads_started":1}
I20260924 16:13:00.845541 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e6536dc57764ab18a71722feeff91ec" candidate_uuid: "14d33685ba3849c78f336923bf49e2d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20193887ad994ccb8f3e9a4e4cad037c" is_pre_election: true
I20260924 16:13:00.845947 5383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e6536dc57764ab18a71722feeff91ec" candidate_uuid: "14d33685ba3849c78f336923bf49e2d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76eabf034e914b95887d119fd3f8faa3" is_pre_election: true
I20260924 16:13:00.846051 5678 raft_consensus.cc:2454] T 4e6536dc57764ab18a71722feeff91ec P 20193887ad994ccb8f3e9a4e4cad037c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14d33685ba3849c78f336923bf49e2d4 in term 0.
I20260924 16:13:00.846436 5383 raft_consensus.cc:2454] T 4e6536dc57764ab18a71722feeff91ec P 76eabf034e914b95887d119fd3f8faa3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14d33685ba3849c78f336923bf49e2d4 in term 0.
I20260924 16:13:00.846728 5468 leader_election.cc:302] T 4e6536dc57764ab18a71722feeff91ec P 14d33685ba3849c78f336923bf49e2d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14d33685ba3849c78f336923bf49e2d4, 20193887ad994ccb8f3e9a4e4cad037c; no voters:
I20260924 16:13:00.847157 5763 raft_consensus.cc:2781] T 4e6536dc57764ab18a71722feeff91ec P 14d33685ba3849c78f336923bf49e2d4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:13:00.847277 5763 raft_consensus.cc:482] T 4e6536dc57764ab18a71722feeff91ec P 14d33685ba3849c78f336923bf49e2d4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:13:00.847426 5763 raft_consensus.cc:3037] T 4e6536dc57764ab18a71722feeff91ec P 14d33685ba3849c78f336923bf49e2d4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:13:00.850342 5763 raft_consensus.cc:504] T 4e6536dc57764ab18a71722feeff91ec P 14d33685ba3849c78f336923bf49e2d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d33685ba3849c78f336923bf49e2d4" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 39763 } } peers { permanent_uuid: "76eabf034e914b95887d119fd3f8faa3" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 40063 } } peers { permanent_uuid: "20193887ad994ccb8f3e9a4e4cad037c" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 34375 } }
I20260924 16:13:00.850956 5763 leader_election.cc:288] T 4e6536dc57764ab18a71722feeff91ec P 14d33685ba3849c78f336923bf49e2d4 [CANDIDATE]: Term 1 election: Requested vote from peers 76eabf034e914b95887d119fd3f8faa3 (127.29.234.1:40063), 20193887ad994ccb8f3e9a4e4cad037c (127.29.234.3:34375)
I20260924 16:13:00.851435 5383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e6536dc57764ab18a71722feeff91ec" candidate_uuid: "14d33685ba3849c78f336923bf49e2d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76eabf034e914b95887d119fd3f8faa3"
I20260924 16:13:00.851629 5383 raft_consensus.cc:3037] T 4e6536dc57764ab18a71722feeff91ec P 76eabf034e914b95887d119fd3f8faa3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:13:00.853884 5383 raft_consensus.cc:2454] T 4e6536dc57764ab18a71722feeff91ec P 76eabf034e914b95887d119fd3f8faa3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14d33685ba3849c78f336923bf49e2d4 in term 1.
I20260924 16:13:00.854223 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e6536dc57764ab18a71722feeff91ec" candidate_uuid: "14d33685ba3849c78f336923bf49e2d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20193887ad994ccb8f3e9a4e4cad037c"
I20260924 16:13:00.854277 5469 leader_election.cc:302] T 4e6536dc57764ab18a71722feeff91ec P 14d33685ba3849c78f336923bf49e2d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14d33685ba3849c78f336923bf49e2d4, 76eabf034e914b95887d119fd3f8faa3; no voters:
I20260924 16:13:00.854454 5678 raft_consensus.cc:3037] T 4e6536dc57764ab18a71722feeff91ec P 20193887ad994ccb8f3e9a4e4cad037c [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:13:00.854592 5763 raft_consensus.cc:2781] T 4e6536dc57764ab18a71722feeff91ec P 14d33685ba3849c78f336923bf49e2d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:13:00.855085 5763 raft_consensus.cc:686] T 4e6536dc57764ab18a71722feeff91ec P 14d33685ba3849c78f336923bf49e2d4 [term 1 LEADER]: Becoming Leader. State: Replica: 14d33685ba3849c78f336923bf49e2d4, State: Running, Role: LEADER
I20260924 16:13:00.855892 5763 consensus_queue.cc:237] T 4e6536dc57764ab18a71722feeff91ec P 14d33685ba3849c78f336923bf49e2d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d33685ba3849c78f336923bf49e2d4" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 39763 } } peers { permanent_uuid: "76eabf034e914b95887d119fd3f8faa3" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 40063 } } peers { permanent_uuid: "20193887ad994ccb8f3e9a4e4cad037c" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 34375 } }
I20260924 16:13:00.856990 5678 raft_consensus.cc:2454] T 4e6536dc57764ab18a71722feeff91ec P 20193887ad994ccb8f3e9a4e4cad037c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14d33685ba3849c78f336923bf49e2d4 in term 1.
I20260924 16:13:00.861243 4447 catalog_manager.cc:5495] T 4e6536dc57764ab18a71722feeff91ec P 14d33685ba3849c78f336923bf49e2d4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 14d33685ba3849c78f336923bf49e2d4 (127.29.234.2). New cstate: current_term: 1 leader_uuid: "14d33685ba3849c78f336923bf49e2d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d33685ba3849c78f336923bf49e2d4" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 39763 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76eabf034e914b95887d119fd3f8faa3" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 40063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20193887ad994ccb8f3e9a4e4cad037c" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 34375 } health_report { overall_health: UNKNOWN } } }
I20260924 16:13:00.908134 5752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:00.902769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55254 (local address 127.29.234.2:39763)
0924 16:13:00.902947 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:00.902952 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:00.903164 (+ 212us) server_negotiation.cc:407] Connection header received
0924 16:13:00.903341 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:00.903346 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:00.903424 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:00.903550 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:13:00.903981 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:00.904680 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:00.906235 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:00.906642 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:00.906741 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:00.907160 (+ 419us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:13:00.907675 (+ 515us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:13:00.907801 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:00.907848 (+ 47us) server_negotiation.cc:299] Negotiation successful
0924 16:13:00.907913 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 16:13:00.916496 5678 raft_consensus.cc:1261] T 4e6536dc57764ab18a71722feeff91ec P 20193887ad994ccb8f3e9a4e4cad037c [term 1 FOLLOWER]: Refusing update from remote peer 14d33685ba3849c78f336923bf49e2d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:13:00.916495 5383 raft_consensus.cc:1261] T 4e6536dc57764ab18a71722feeff91ec P 76eabf034e914b95887d119fd3f8faa3 [term 1 FOLLOWER]: Refusing update from remote peer 14d33685ba3849c78f336923bf49e2d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:13:00.917580 5763 consensus_queue.cc:1035] T 4e6536dc57764ab18a71722feeff91ec P 14d33685ba3849c78f336923bf49e2d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76eabf034e914b95887d119fd3f8faa3" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 40063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:13:00.917878 5769 consensus_queue.cc:1035] T 4e6536dc57764ab18a71722feeff91ec P 14d33685ba3849c78f336923bf49e2d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20193887ad994ccb8f3e9a4e4cad037c" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 34375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:13:00.931258 5773 mvcc.cc:205] Tried to move back new op lower bound from 7332931096223997952 to 7332931095989731328. Current Snapshot: MvccSnapshot[applied={T|T < 7332931096223997952}]
I20260924 16:13:00.954828 4447 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:13:00.955124 4447 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:13:00.962543 4447 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d3639f056333442f859ccc6c600711ac: Sending DeleteTablet for 3 replicas of tablet 4e6536dc57764ab18a71722feeff91ec
I20260924 16:13:00.963995 5513 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e6536dc57764ab18a71722feeff91ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:13:00 UTC) from {username='oryx', principal='oryx/127.29.234.62@KRBTEST.COM'} at 127.0.0.1:55238
I20260924 16:13:00.965548 5362 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e6536dc57764ab18a71722feeff91ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:13:00 UTC) from {username='oryx', principal='oryx/127.29.234.62@KRBTEST.COM'} at 127.0.0.1:50298
I20260924 16:13:00.965574 5657 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e6536dc57764ab18a71722feeff91ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:13:00 UTC) from {username='oryx', principal='oryx/127.29.234.62@KRBTEST.COM'} at 127.0.0.1:47356
I20260924 16:13:00.973729 5782 tablet_replica.cc:330] T 4e6536dc57764ab18a71722feeff91ec P 14d33685ba3849c78f336923bf49e2d4: stopping tablet replica
I20260924 16:13:00.974339 5782 raft_consensus.cc:2229] T 4e6536dc57764ab18a71722feeff91ec P 14d33685ba3849c78f336923bf49e2d4 [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:13:00.975170 5782 raft_consensus.cc:2258] T 4e6536dc57764ab18a71722feeff91ec P 14d33685ba3849c78f336923bf49e2d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:13:00.977813 5783 tablet_replica.cc:330] T 4e6536dc57764ab18a71722feeff91ec P 20193887ad994ccb8f3e9a4e4cad037c: stopping tablet replica
I20260924 16:13:00.977867 5782 ts_tablet_manager.cc:1905] T 4e6536dc57764ab18a71722feeff91ec P 14d33685ba3849c78f336923bf49e2d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:13:00.978305 5783 raft_consensus.cc:2229] T 4e6536dc57764ab18a71722feeff91ec P 20193887ad994ccb8f3e9a4e4cad037c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:13:00.978888 5783 raft_consensus.cc:2258] T 4e6536dc57764ab18a71722feeff91ec P 20193887ad994ccb8f3e9a4e4cad037c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:13:00.979720 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 4412
I20260924 16:13:00.981010 4477 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:13:00.987159 5782 ts_tablet_manager.cc:1918] T 4e6536dc57764ab18a71722feeff91ec P 14d33685ba3849c78f336923bf49e2d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:13:00.987375 5782 log.cc:1196] T 4e6536dc57764ab18a71722feeff91ec P 14d33685ba3849c78f336923bf49e2d4: Deleting WAL directory at /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-1/wal/wals/4e6536dc57764ab18a71722feeff91ec
I20260924 16:13:00.988039 5782 ts_tablet_manager.cc:1939] T 4e6536dc57764ab18a71722feeff91ec P 14d33685ba3849c78f336923bf49e2d4: Deleting consensus metadata
I20260924 16:13:00.993952 5784 tablet_replica.cc:330] T 4e6536dc57764ab18a71722feeff91ec P 76eabf034e914b95887d119fd3f8faa3: stopping tablet replica
I20260924 16:13:00.994462 5784 raft_consensus.cc:2229] T 4e6536dc57764ab18a71722feeff91ec P 76eabf034e914b95887d119fd3f8faa3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:13:00.995023 5784 raft_consensus.cc:2258] T 4e6536dc57764ab18a71722feeff91ec P 76eabf034e914b95887d119fd3f8faa3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:13:00.997256 5784 ts_tablet_manager.cc:1905] T 4e6536dc57764ab18a71722feeff91ec P 76eabf034e914b95887d119fd3f8faa3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:13:01.000134 5783 ts_tablet_manager.cc:1905] T 4e6536dc57764ab18a71722feeff91ec P 20193887ad994ccb8f3e9a4e4cad037c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:13:01.004868 5784 ts_tablet_manager.cc:1918] T 4e6536dc57764ab18a71722feeff91ec P 76eabf034e914b95887d119fd3f8faa3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:13:01.005030 5784 log.cc:1196] T 4e6536dc57764ab18a71722feeff91ec P 76eabf034e914b95887d119fd3f8faa3: Deleting WAL directory at /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-0/wal/wals/4e6536dc57764ab18a71722feeff91ec
I20260924 16:13:01.008029 5783 ts_tablet_manager.cc:1918] T 4e6536dc57764ab18a71722feeff91ec P 20193887ad994ccb8f3e9a4e4cad037c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:13:01.008231 5783 log.cc:1196] T 4e6536dc57764ab18a71722feeff91ec P 20193887ad994ccb8f3e9a4e4cad037c: Deleting WAL directory at /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-2/wal/wals/4e6536dc57764ab18a71722feeff91ec
I20260924 16:13:01.008801 5783 ts_tablet_manager.cc:1939] T 4e6536dc57764ab18a71722feeff91ec P 20193887ad994ccb8f3e9a4e4cad037c: Deleting consensus metadata
I20260924 16:13:01.012864 5786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:00.968688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52136 (local address 127.29.234.61:42395)
0924 16:13:00.993868 (+ 25180us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:00.993878 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:00.993908 (+ 30us) server_negotiation.cc:407] Connection header received
0924 16:13:00.993997 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:00.994006 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:00.994210 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:00.994375 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:00.996901 (+ 2526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:00.998429 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:01.000594 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:01.001198 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:01.001317 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:01.005052 (+ 3735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:13:01.005059 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:13:01.005065 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:13:01.005205 (+ 140us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:13:01.009646 (+ 4441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:01.010700 (+ 1054us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:01.010706 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:01.010710 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:01.010875 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:01.011295 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:01.011302 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:01.011306 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:01.011613 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:13:01.011882 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:01.012203 (+ 321us) server_negotiation.cc:299] Negotiation successful
0924 16:13:01.012478 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24973,"thread_start_us":24837,"threads_started":1}
I20260924 16:13:01.017992 5784 ts_tablet_manager.cc:1939] T 4e6536dc57764ab18a71722feeff91ec P 76eabf034e914b95887d119fd3f8faa3: Deleting consensus metadata
I20260924 16:13:01.026753 5787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:00.976600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35548 (local address 127.29.234.60:45643)
0924 16:13:00.994901 (+ 18301us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:00.994907 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:01.013897 (+ 18990us) server_negotiation.cc:407] Connection header received
0924 16:13:01.014052 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:01.014058 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:01.014148 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:01.014271 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:01.015751 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:01.016515 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:01.017807 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:01.018318 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:01.018456 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:01.021162 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:13:01.021168 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:13:01.021172 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:13:01.021203 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:13:01.023955 (+ 2752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:01.024688 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:01.024693 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:01.024695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:01.024821 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:01.025183 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:01.025191 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:01.025194 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:01.025539 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:13:01.025798 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:01.026116 (+ 318us) server_negotiation.cc:299] Negotiation successful
0924 16:13:01.026348 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18181,"thread_start_us":18082,"threads_started":1}
I20260924 16:13:01.146512 4434 catalog_manager.cc:4841] TS 76eabf034e914b95887d119fd3f8faa3 (127.29.234.1:40063): tablet 4e6536dc57764ab18a71722feeff91ec (table test-table [id=e338f019f6f647fab6f690debfc0c753]) successfully deleted
I20260924 16:13:01.147950 4433 catalog_manager.cc:4841] TS 20193887ad994ccb8f3e9a4e4cad037c (127.29.234.3:34375): tablet 4e6536dc57764ab18a71722feeff91ec (table test-table [id=e338f019f6f647fab6f690debfc0c753]) successfully deleted
I20260924 16:13:01.148419 4432 catalog_manager.cc:4841] TS 14d33685ba3849c78f336923bf49e2d4 (127.29.234.2:39763): tablet 4e6536dc57764ab18a71722feeff91ec (table test-table [id=e338f019f6f647fab6f690debfc0c753]) successfully deleted
I20260924 16:13:01.148512 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 4412
I20260924 16:13:01.204638 5786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:01.181765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52150 (local address 127.29.234.61:42395)
0924 16:13:01.181967 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:01.181973 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:01.187012 (+ 5039us) server_negotiation.cc:407] Connection header received
0924 16:13:01.187704 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:01.187714 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:01.187841 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:01.187937 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:01.191884 (+ 3947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:01.192817 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:01.194347 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:01.194803 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:01.194969 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:01.197656 (+ 2687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:13:01.197662 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:13:01.197672 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:13:01.197704 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:13:01.200333 (+ 2629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:01.200944 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:01.200949 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:01.200953 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:01.201073 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:01.201685 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:01.201689 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:01.201691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:01.201951 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:13:01.202199 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:01.202582 (+ 383us) server_negotiation.cc:299] Negotiation successful
0924 16:13:01.204307 (+ 1725us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 16:13:01.206749 5787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:01.181722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35554 (local address 127.29.234.60:45643)
0924 16:13:01.181961 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:01.181968 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:01.190821 (+ 8853us) server_negotiation.cc:407] Connection header received
0924 16:13:01.191150 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:01.191158 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:01.191250 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:01.191360 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:01.194000 (+ 2640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:01.195086 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:01.197053 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:01.197708 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:01.197810 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:01.200740 (+ 2930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:13:01.200747 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:13:01.200751 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:13:01.200796 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:13:01.203981 (+ 3185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:01.204558 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:01.204564 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:01.204573 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:01.204726 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:01.205239 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:01.205244 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:01.205247 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:01.205595 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:13:01.206043 (+ 448us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:01.206205 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 16:13:01.206421 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 16:13:01.940006 5796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:01.939060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:40099 (local address 127.29.234.2:58371)
0924 16:13:01.939660 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:01.939778 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:40099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":349,"threads_started":1}
W20260924 16:13:01.940436 5581 heartbeater.cc:644] Failed to heartbeat to 127.29.234.62:40099 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.29.234.62:40099: connect: Connection refused (error 111)
I20260924 16:13:01.949801 5797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:01.947902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:40099 (local address 127.29.234.3:46967)
0924 16:13:01.949496 (+ 1594us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:01.949640 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:40099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1404,"thread_start_us":1193,"threads_started":1}
I20260924 16:13:01.950030 5798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:01.948682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:40099 (local address 127.29.234.1:56359)
0924 16:13:01.949689 (+ 1007us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:01.949824 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:40099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":886,"thread_start_us":782,"threads_started":1}
W20260924 16:13:01.950359 5724 heartbeater.cc:644] Failed to heartbeat to 127.29.234.62:40099 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.29.234.62:40099: connect: Connection refused (error 111)
W20260924 16:13:01.950538 5429 heartbeater.cc:644] Failed to heartbeat to 127.29.234.62:40099 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.29.234.62:40099: connect: Connection refused (error 111)
I20260924 16:13:02.462846 5799 raft_consensus.cc:482] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3639f056333442f859ccc6c600711ac)
I20260924 16:13:02.463088 5799 raft_consensus.cc:504] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d3639f056333442f859ccc6c600711ac" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 40099 } } peers { permanent_uuid: "d44fd4dbe8224ecf93501b1d97a76304" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 42395 } attrs { promote: false } } peers { permanent_uuid: "7785d6b6f5bf4dd5a2667857b24dcf8a" member_type: VOTER last_known_addr { host: "127.29.234.60" port: 45643 } attrs { promote: false } }
I20260924 16:13:02.463884 5800 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7785d6b6f5bf4dd5a2667857b24dcf8a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3639f056333442f859ccc6c600711ac)
I20260924 16:13:02.464068 5799 leader_election.cc:288] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d3639f056333442f859ccc6c600711ac (127.29.234.62:40099), 7785d6b6f5bf4dd5a2667857b24dcf8a (127.29.234.60:45643)
I20260924 16:13:02.464308 5800 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7785d6b6f5bf4dd5a2667857b24dcf8a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d3639f056333442f859ccc6c600711ac" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 40099 } } peers { permanent_uuid: "d44fd4dbe8224ecf93501b1d97a76304" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 42395 } attrs { promote: false } } peers { permanent_uuid: "7785d6b6f5bf4dd5a2667857b24dcf8a" member_type: VOTER last_known_addr { host: "127.29.234.60" port: 45643 } attrs { promote: false } }
I20260924 16:13:02.465973 5801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:02.464322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:40099 (local address 127.0.0.1:37932)
0924 16:13:02.465330 (+ 1008us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:02.465606 (+ 276us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:40099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":835,"thread_start_us":320,"threads_started":1}
W20260924 16:13:02.466312 5042 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.234.62:40099: connect: Connection refused (error 111)
I20260924 16:13:02.468009 5801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:02.467499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:40099 (local address 127.0.0.1:37952)
0924 16:13:02.467665 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:02.467819 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:40099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 16:13:02.468326 5042 leader_election.cc:334] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d3639f056333442f859ccc6c600711ac (127.29.234.62:40099): Network error: Client connection negotiation failed: client connection to 127.29.234.62:40099: connect: Connection refused (error 111)
I20260924 16:13:02.469486 5800 leader_election.cc:288] T 00000000000000000000000000000000 P 7785d6b6f5bf4dd5a2667857b24dcf8a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d3639f056333442f859ccc6c600711ac (127.29.234.62:40099), d44fd4dbe8224ecf93501b1d97a76304 (127.29.234.61:42395)
I20260924 16:13:02.471240 5805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:02.467229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:40099 (local address 127.0.0.1:37948)
0924 16:13:02.470682 (+ 3453us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:02.470931 (+ 249us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:40099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3313,"thread_start_us":3186,"threads_started":1}
W20260924 16:13:02.471614 5233 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.234.62:40099: connect: Connection refused (error 111)
I20260924 16:13:02.473282 5805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:02.472925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:40099 (local address 127.0.0.1:37954)
0924 16:13:02.473067 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:02.473156 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:40099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260924 16:13:02.473476 5233 leader_election.cc:334] T 00000000000000000000000000000000 P 7785d6b6f5bf4dd5a2667857b24dcf8a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d3639f056333442f859ccc6c600711ac (127.29.234.62:40099): Network error: Client connection negotiation failed: client connection to 127.29.234.62:40099: connect: Connection refused (error 111)
I20260924 16:13:02.473896 5803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:02.464595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57488 (local address 127.29.234.60:45643)
0924 16:13:02.465579 (+ 984us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:02.465585 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:02.465604 (+ 19us) server_negotiation.cc:407] Connection header received
0924 16:13:02.465672 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:02.465676 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:02.465753 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:02.465864 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:13:02.466661 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:02.467899 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:02.470926 (+ 3027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:02.472497 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:02.472634 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:02.472774 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:02.473428 (+ 654us) server_negotiation.cc:299] Negotiation successful
0924 16:13:02.473584 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":857,"thread_start_us":689,"threads_started":1}
I20260924 16:13:02.474174 5802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:02.464316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.60:45643 (local address 127.0.0.1:57488)
0924 16:13:02.464960 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:02.464989 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:02.465140 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:02.465929 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:02.465934 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:02.465975 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:13:02.466420 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:02.466435 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:02.468059 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:02.468066 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:02.470667 (+ 2601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:02.470689 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:02.472663 (+ 1974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:02.472669 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:02.473195 (+ 526us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:02.473237 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 16:13:02.473375 (+ 138us) client_negotiation.cc:240] Negotiation successful
0924 16:13:02.473870 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":679,"threads_started":1}
I20260924 16:13:02.474879 5257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d44fd4dbe8224ecf93501b1d97a76304" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7785d6b6f5bf4dd5a2667857b24dcf8a" is_pre_election: true
I20260924 16:13:02.475201 5257 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7785d6b6f5bf4dd5a2667857b24dcf8a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d44fd4dbe8224ecf93501b1d97a76304 in term 1.
I20260924 16:13:02.475682 5043 leader_election.cc:302] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7785d6b6f5bf4dd5a2667857b24dcf8a, d44fd4dbe8224ecf93501b1d97a76304; no voters: d3639f056333442f859ccc6c600711ac
I20260924 16:13:02.476059 5799 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 16:13:02.476187 5799 raft_consensus.cc:482] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d3639f056333442f859ccc6c600711ac)
I20260924 16:13:02.476293 5799 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304 [term 1 FOLLOWER]: Advancing to term 2
I20260924 16:13:02.477917 5806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:02.468020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.61:42395 (local address 127.0.0.1:50818)
0924 16:13:02.470991 (+ 2971us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:02.471016 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:02.471146 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:02.471645 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:02.471651 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:02.471686 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:13:02.472148 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:02.472159 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:02.473593 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:02.473600 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:02.475357 (+ 1757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:02.475374 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:02.477016 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:02.477024 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:02.477286 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:02.477319 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 16:13:02.477443 (+ 124us) client_negotiation.cc:240] Negotiation successful
0924 16:13:02.477575 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2860,"thread_start_us":2678,"threads_started":1}
I20260924 16:13:02.477954 5807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:02.468352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50818 (local address 127.29.234.61:42395)
0924 16:13:02.470276 (+ 1924us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:02.470285 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:02.471110 (+ 825us) server_negotiation.cc:407] Connection header received
0924 16:13:02.471284 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:02.471290 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:02.471363 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:02.471464 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:13:02.472414 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:02.473476 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:02.475583 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:02.476823 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:02.476956 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:02.477080 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:02.477470 (+ 390us) server_negotiation.cc:299] Negotiation successful
0924 16:13:02.477572 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1735,"thread_start_us":1500,"threads_started":1}
I20260924 16:13:02.478786 5067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7785d6b6f5bf4dd5a2667857b24dcf8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d44fd4dbe8224ecf93501b1d97a76304" is_pre_election: true
I20260924 16:13:02.479197 5799 raft_consensus.cc:504] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d3639f056333442f859ccc6c600711ac" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 40099 } } peers { permanent_uuid: "d44fd4dbe8224ecf93501b1d97a76304" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 42395 } attrs { promote: false } } peers { permanent_uuid: "7785d6b6f5bf4dd5a2667857b24dcf8a" member_type: VOTER last_known_addr { host: "127.29.234.60" port: 45643 } attrs { promote: false } }
I20260924 16:13:02.479592 5067 raft_consensus.cc:2379] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7785d6b6f5bf4dd5a2667857b24dcf8a in current term 2: Already voted for candidate d44fd4dbe8224ecf93501b1d97a76304 in this term.
I20260924 16:13:02.479857 5799 leader_election.cc:288] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304 [CANDIDATE]: Term 2 election: Requested vote from peers d3639f056333442f859ccc6c600711ac (127.29.234.62:40099), 7785d6b6f5bf4dd5a2667857b24dcf8a (127.29.234.60:45643)
I20260924 16:13:02.480342 5802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:02.479961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:40099 (local address 127.0.0.1:37970)
0924 16:13:02.480151 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:02.480243 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:40099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260924 16:13:02.480305 5257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d44fd4dbe8224ecf93501b1d97a76304" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7785d6b6f5bf4dd5a2667857b24dcf8a"
I20260924 16:13:02.480376 5231 leader_election.cc:302] T 00000000000000000000000000000000 P 7785d6b6f5bf4dd5a2667857b24dcf8a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7785d6b6f5bf4dd5a2667857b24dcf8a; no voters: d3639f056333442f859ccc6c600711ac, d44fd4dbe8224ecf93501b1d97a76304
I20260924 16:13:02.480526 5257 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7785d6b6f5bf4dd5a2667857b24dcf8a [term 1 FOLLOWER]: Advancing to term 2
I20260924 16:13:02.482223 5802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:02.481900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:40099 (local address 127.0.0.1:37984)
0924 16:13:02.482035 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:02.482120 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:40099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 16:13:02.482381 5042 leader_election.cc:334] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer d3639f056333442f859ccc6c600711ac (127.29.234.62:40099): Network error: Client connection negotiation failed: client connection to 127.29.234.62:40099: connect: Connection refused (error 111)
I20260924 16:13:02.482942 5257 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7785d6b6f5bf4dd5a2667857b24dcf8a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d44fd4dbe8224ecf93501b1d97a76304 in term 2.
I20260924 16:13:02.483115 5800 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 7785d6b6f5bf4dd5a2667857b24dcf8a [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 16:13:02.483343 5043 leader_election.cc:302] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7785d6b6f5bf4dd5a2667857b24dcf8a, d44fd4dbe8224ecf93501b1d97a76304; no voters: d3639f056333442f859ccc6c600711ac
I20260924 16:13:02.483743 5799 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 16:13:02.484606 5799 raft_consensus.cc:686] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304 [term 2 LEADER]: Becoming Leader. State: Replica: d44fd4dbe8224ecf93501b1d97a76304, State: Running, Role: LEADER
I20260924 16:13:02.484992 5799 consensus_queue.cc:237] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d3639f056333442f859ccc6c600711ac" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 40099 } } peers { permanent_uuid: "d44fd4dbe8224ecf93501b1d97a76304" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 42395 } attrs { promote: false } } peers { permanent_uuid: "7785d6b6f5bf4dd5a2667857b24dcf8a" member_type: VOTER last_known_addr { host: "127.29.234.60" port: 45643 } attrs { promote: false } }
I20260924 16:13:02.487708 5809 sys_catalog.cc:455] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d44fd4dbe8224ecf93501b1d97a76304. Latest consensus state: current_term: 2 leader_uuid: "d44fd4dbe8224ecf93501b1d97a76304" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d3639f056333442f859ccc6c600711ac" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 40099 } } peers { permanent_uuid: "d44fd4dbe8224ecf93501b1d97a76304" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 42395 } attrs { promote: false } } peers { permanent_uuid: "7785d6b6f5bf4dd5a2667857b24dcf8a" member_type: VOTER last_known_addr { host: "127.29.234.60" port: 45643 } attrs { promote: false } } }
I20260924 16:13:02.488099 5809 sys_catalog.cc:458] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304 [sys.catalog]: This master's current role is: LEADER
I20260924 16:13:02.488924 5811 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:13:02.493281 5811 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:13:02.494134 5811 catalog_manager.cc:1259] Loaded cluster ID: e95643b1533745a68073f27a96aa4b1a
I20260924 16:13:02.494240 5811 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:13:02.495393 5811 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:13:02.496208 5811 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304: Loaded TSK: 0
I20260924 16:13:02.497064 5811 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:13:02.546696 5057 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52150:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:13:02.553802 5057 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:13:02.557982 5802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:02.557524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:40099 (local address 127.0.0.1:38004)
0924 16:13:02.557696 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:02.557791 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:40099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260924 16:13:02.558005 5257 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7785d6b6f5bf4dd5a2667857b24dcf8a [term 2 FOLLOWER]: Refusing update from remote peer d44fd4dbe8224ecf93501b1d97a76304: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260924 16:13:02.558692 5809 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7785d6b6f5bf4dd5a2667857b24dcf8a" member_type: VOTER last_known_addr { host: "127.29.234.60" port: 45643 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260924 16:13:02.559088 5802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:02.558623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:40099 (local address 127.0.0.1:38010)
0924 16:13:02.558831 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:02.558921 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:40099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260924 16:13:02.559340 5042 consensus_peers.cc:487] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304 -> Peer d3639f056333442f859ccc6c600711ac (127.29.234.62:40099): Couldn't send request to peer d3639f056333442f859ccc6c600711ac. Status: Network error: Client connection negotiation failed: client connection to 127.29.234.62:40099: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 16:13:02.561583 5800 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7785d6b6f5bf4dd5a2667857b24dcf8a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d44fd4dbe8224ecf93501b1d97a76304. Latest consensus state: current_term: 2 leader_uuid: "d44fd4dbe8224ecf93501b1d97a76304" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d3639f056333442f859ccc6c600711ac" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 40099 } } peers { permanent_uuid: "d44fd4dbe8224ecf93501b1d97a76304" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 42395 } attrs { promote: false } } peers { permanent_uuid: "7785d6b6f5bf4dd5a2667857b24dcf8a" member_type: VOTER last_known_addr { host: "127.29.234.60" port: 45643 } attrs { promote: false } } }
I20260924 16:13:02.561851 5800 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7785d6b6f5bf4dd5a2667857b24dcf8a [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:13:02.563921 5809 sys_catalog.cc:455] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "d44fd4dbe8224ecf93501b1d97a76304" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d3639f056333442f859ccc6c600711ac" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 40099 } } peers { permanent_uuid: "d44fd4dbe8224ecf93501b1d97a76304" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 42395 } attrs { promote: false } } peers { permanent_uuid: "7785d6b6f5bf4dd5a2667857b24dcf8a" member_type: VOTER last_known_addr { host: "127.29.234.60" port: 45643 } attrs { promote: false } } }
I20260924 16:13:02.563998 5800 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7785d6b6f5bf4dd5a2667857b24dcf8a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "d44fd4dbe8224ecf93501b1d97a76304" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d3639f056333442f859ccc6c600711ac" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 40099 } } peers { permanent_uuid: "d44fd4dbe8224ecf93501b1d97a76304" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 42395 } attrs { promote: false } } peers { permanent_uuid: "7785d6b6f5bf4dd5a2667857b24dcf8a" member_type: VOTER last_known_addr { host: "127.29.234.60" port: 45643 } attrs { promote: false } } }
I20260924 16:13:02.564213 5800 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7785d6b6f5bf4dd5a2667857b24dcf8a [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:13:02.564218 5809 sys_catalog.cc:458] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304 [sys.catalog]: This master's current role is: LEADER
I20260924 16:13:02.582805 5819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:02.573834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40838 (local address 127.29.234.2:39763)
0924 16:13:02.574365 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:02.574373 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:02.574394 (+ 21us) server_negotiation.cc:407] Connection header received
0924 16:13:02.574467 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:02.574470 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:02.574553 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:02.574659 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:13:02.575420 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:02.576677 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:02.580388 (+ 3711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:02.581612 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:02.581740 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:02.581848 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:02.582352 (+ 504us) server_negotiation.cc:299] Negotiation successful
0924 16:13:02.582498 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":221,"threads_started":1}
I20260924 16:13:02.583024 5801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:02.573674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.2:39763 (local address 127.0.0.1:40838)
0924 16:13:02.573834 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:02.573851 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:02.573972 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:02.574682 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:02.574690 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:02.574723 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:13:02.575182 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:02.575202 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:02.576876 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:02.576885 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:02.580195 (+ 3310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:02.580215 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:02.581878 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:02.581887 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:02.582142 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:02.582196 (+ 54us) client_negotiation.cc:769] Sending connection context
0924 16:13:02.582313 (+ 117us) client_negotiation.cc:240] Negotiation successful
0924 16:13:02.582509 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260924 16:13:02.584028 5513 tablet_service.cc:1467] Processing CreateTablet for tablet fd2597f606e649749ac12252697762a9 (DEFAULT_TABLE table=test-table [id=00b23a5660f048709da6c3e85b884d00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:13:02.584640 5513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd2597f606e649749ac12252697762a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:13:02.584805 5820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:02.574740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:40063 (local address 127.0.0.1:45900)
0924 16:13:02.575405 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:02.575422 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:02.575741 (+ 319us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:02.578198 (+ 2457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:02.578202 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:02.578226 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:13:02.578552 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:02.578560 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:02.579909 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:02.579914 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:02.581965 (+ 2051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:02.581977 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:02.583733 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:02.583739 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:02.584208 (+ 469us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:02.584240 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 16:13:02.584405 (+ 165us) client_negotiation.cc:240] Negotiation successful
0924 16:13:02.584527 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":691,"threads_started":1}
I20260924 16:13:02.585668 5821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:02.575206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45900 (local address 127.29.234.1:40063)
0924 16:13:02.577867 (+ 2661us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:02.577872 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:02.577887 (+ 15us) server_negotiation.cc:407] Connection header received
0924 16:13:02.577945 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:02.577948 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:02.578003 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:02.578169 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:13:02.578748 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:02.579747 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:02.582113 (+ 2366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:02.583250 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:02.585129 (+ 1879us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:02.585227 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:02.585276 (+ 49us) server_negotiation.cc:299] Negotiation successful
0924 16:13:02.585446 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2556,"thread_start_us":2455,"threads_started":1}
I20260924 16:13:02.586465 5362 tablet_service.cc:1467] Processing CreateTablet for tablet fd2597f606e649749ac12252697762a9 (DEFAULT_TABLE table=test-table [id=00b23a5660f048709da6c3e85b884d00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:13:02.586874 5362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd2597f606e649749ac12252697762a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:13:02.593130 5822 tablet_bootstrap.cc:492] T fd2597f606e649749ac12252697762a9 P 14d33685ba3849c78f336923bf49e2d4: Bootstrap starting.
I20260924 16:13:02.595412 5802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:02.573300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.3:34375 (local address 127.0.0.1:48956)
0924 16:13:02.573469 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:02.573492 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:02.573606 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:02.590012 (+ 16406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:02.590016 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:02.590039 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:13:02.590179 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:02.590186 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:02.591542 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:02.591549 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:02.593421 (+ 1872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:02.593436 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:02.594628 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:02.594634 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:02.594817 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:02.594859 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 16:13:02.594988 (+ 129us) client_negotiation.cc:240] Negotiation successful
0924 16:13:02.595132 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 16:13:02.596118 5823 tablet_bootstrap.cc:492] T fd2597f606e649749ac12252697762a9 P 76eabf034e914b95887d119fd3f8faa3: Bootstrap starting.
I20260924 16:13:02.596341 5818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:02.573459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48956 (local address 127.29.234.3:34375)
0924 16:13:02.589686 (+ 16227us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:02.589693 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:02.589713 (+ 20us) server_negotiation.cc:407] Connection header received
0924 16:13:02.589785 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:02.589791 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:02.589863 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:02.589972 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:13:02.590296 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:02.591377 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:02.593692 (+ 2315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:02.594448 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:02.594603 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:02.594781 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:02.595890 (+ 1109us) server_negotiation.cc:299] Negotiation successful
0924 16:13:02.596043 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16093,"thread_start_us":16636,"threads_started":1}
I20260924 16:13:02.596603 5822 tablet_bootstrap.cc:654] T fd2597f606e649749ac12252697762a9 P 14d33685ba3849c78f336923bf49e2d4: Neither blocks nor log segments found. Creating new log.
I20260924 16:13:02.597061 5657 tablet_service.cc:1467] Processing CreateTablet for tablet fd2597f606e649749ac12252697762a9 (DEFAULT_TABLE table=test-table [id=00b23a5660f048709da6c3e85b884d00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:13:02.597580 5657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd2597f606e649749ac12252697762a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:13:02.597918 5823 tablet_bootstrap.cc:654] T fd2597f606e649749ac12252697762a9 P 76eabf034e914b95887d119fd3f8faa3: Neither blocks nor log segments found. Creating new log.
I20260924 16:13:02.598799 5822 tablet_bootstrap.cc:492] T fd2597f606e649749ac12252697762a9 P 14d33685ba3849c78f336923bf49e2d4: No bootstrap required, opened a new log
I20260924 16:13:02.599045 5822 ts_tablet_manager.cc:1397] T fd2597f606e649749ac12252697762a9 P 14d33685ba3849c78f336923bf49e2d4: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260924 16:13:02.599810 5822 raft_consensus.cc:348] T fd2597f606e649749ac12252697762a9 P 14d33685ba3849c78f336923bf49e2d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20193887ad994ccb8f3e9a4e4cad037c" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 34375 } } peers { permanent_uuid: "14d33685ba3849c78f336923bf49e2d4" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 39763 } } peers { permanent_uuid: "76eabf034e914b95887d119fd3f8faa3" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 40063 } }
I20260924 16:13:02.600016 5822 raft_consensus.cc:374] T fd2597f606e649749ac12252697762a9 P 14d33685ba3849c78f336923bf49e2d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:13:02.600039 5823 tablet_bootstrap.cc:492] T fd2597f606e649749ac12252697762a9 P 76eabf034e914b95887d119fd3f8faa3: No bootstrap required, opened a new log
I20260924 16:13:02.600112 5822 raft_consensus.cc:729] T fd2597f606e649749ac12252697762a9 P 14d33685ba3849c78f336923bf49e2d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14d33685ba3849c78f336923bf49e2d4, State: Initialized, Role: FOLLOWER
I20260924 16:13:02.600157 5823 ts_tablet_manager.cc:1397] T fd2597f606e649749ac12252697762a9 P 76eabf034e914b95887d119fd3f8faa3: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260924 16:13:02.600363 5822 consensus_queue.cc:260] T fd2597f606e649749ac12252697762a9 P 14d33685ba3849c78f336923bf49e2d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20193887ad994ccb8f3e9a4e4cad037c" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 34375 } } peers { permanent_uuid: "14d33685ba3849c78f336923bf49e2d4" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 39763 } } peers { permanent_uuid: "76eabf034e914b95887d119fd3f8faa3" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 40063 } }
I20260924 16:13:02.600883 5823 raft_consensus.cc:348] T fd2597f606e649749ac12252697762a9 P 76eabf034e914b95887d119fd3f8faa3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20193887ad994ccb8f3e9a4e4cad037c" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 34375 } } peers { permanent_uuid: "14d33685ba3849c78f336923bf49e2d4" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 39763 } } peers { permanent_uuid: "76eabf034e914b95887d119fd3f8faa3" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 40063 } }
I20260924 16:13:02.601141 5823 raft_consensus.cc:374] T fd2597f606e649749ac12252697762a9 P 76eabf034e914b95887d119fd3f8faa3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:13:02.601241 5823 raft_consensus.cc:729] T fd2597f606e649749ac12252697762a9 P 76eabf034e914b95887d119fd3f8faa3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76eabf034e914b95887d119fd3f8faa3, State: Initialized, Role: FOLLOWER
I20260924 16:13:02.601482 5823 consensus_queue.cc:260] T fd2597f606e649749ac12252697762a9 P 76eabf034e914b95887d119fd3f8faa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20193887ad994ccb8f3e9a4e4cad037c" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 34375 } } peers { permanent_uuid: "14d33685ba3849c78f336923bf49e2d4" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 39763 } } peers { permanent_uuid: "76eabf034e914b95887d119fd3f8faa3" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 40063 } }
I20260924 16:13:02.603526 5822 ts_tablet_manager.cc:1428] T fd2597f606e649749ac12252697762a9 P 14d33685ba3849c78f336923bf49e2d4: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260924 16:13:02.608560 5823 ts_tablet_manager.cc:1428] T fd2597f606e649749ac12252697762a9 P 76eabf034e914b95887d119fd3f8faa3: Time spent starting tablet: real 0.008s user 0.001s sys 0.008s
I20260924 16:13:02.609896 5582 heartbeater.cc:503] Master 127.29.234.61:42395 was elected leader, sending a full tablet report...
I20260924 16:13:02.612591 5827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:02.601454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:40099 (local address 127.29.234.2:59105)
0924 16:13:02.612135 (+ 10681us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:02.612304 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:40099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":10546,"thread_start_us":10349,"threads_started":1}
I20260924 16:13:02.613587 5430 heartbeater.cc:503] Master 127.29.234.61:42395 was elected leader, sending a full tablet report...
I20260924 16:13:02.614224 5827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:02.613310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:40099 (local address 127.29.234.2:47785)
0924 16:13:02.613982 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:02.614066 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:40099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":578,"threads_started":1}
I20260924 16:13:02.614244 5829 tablet_bootstrap.cc:492] T fd2597f606e649749ac12252697762a9 P 20193887ad994ccb8f3e9a4e4cad037c: Bootstrap starting.
I20260924 16:13:02.616207 5829 tablet_bootstrap.cc:654] T fd2597f606e649749ac12252697762a9 P 20193887ad994ccb8f3e9a4e4cad037c: Neither blocks nor log segments found. Creating new log.
I20260924 16:13:02.618244 5830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:02.611413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:40099 (local address 127.29.234.1:40107)
0924 16:13:02.617799 (+ 6386us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:02.617981 (+ 182us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:40099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6243,"thread_start_us":6133,"threads_started":1}
I20260924 16:13:02.618527 5829 tablet_bootstrap.cc:492] T fd2597f606e649749ac12252697762a9 P 20193887ad994ccb8f3e9a4e4cad037c: No bootstrap required, opened a new log
I20260924 16:13:02.618677 5829 ts_tablet_manager.cc:1397] T fd2597f606e649749ac12252697762a9 P 20193887ad994ccb8f3e9a4e4cad037c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 16:13:02.619266 5830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:02.618905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:40099 (local address 127.29.234.1:39491)
0924 16:13:02.619058 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:02.619142 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:40099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260924 16:13:02.619503 5829 raft_consensus.cc:348] T fd2597f606e649749ac12252697762a9 P 20193887ad994ccb8f3e9a4e4cad037c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20193887ad994ccb8f3e9a4e4cad037c" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 34375 } } peers { permanent_uuid: "14d33685ba3849c78f336923bf49e2d4" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 39763 } } peers { permanent_uuid: "76eabf034e914b95887d119fd3f8faa3" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 40063 } }
I20260924 16:13:02.619699 5829 raft_consensus.cc:374] T fd2597f606e649749ac12252697762a9 P 20193887ad994ccb8f3e9a4e4cad037c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:13:02.619767 5829 raft_consensus.cc:729] T fd2597f606e649749ac12252697762a9 P 20193887ad994ccb8f3e9a4e4cad037c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20193887ad994ccb8f3e9a4e4cad037c, State: Initialized, Role: FOLLOWER
I20260924 16:13:02.620012 5829 consensus_queue.cc:260] T fd2597f606e649749ac12252697762a9 P 20193887ad994ccb8f3e9a4e4cad037c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20193887ad994ccb8f3e9a4e4cad037c" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 34375 } } peers { permanent_uuid: "14d33685ba3849c78f336923bf49e2d4" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 39763 } } peers { permanent_uuid: "76eabf034e914b95887d119fd3f8faa3" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 40063 } }
I20260924 16:13:02.620690 5829 ts_tablet_manager.cc:1428] T fd2597f606e649749ac12252697762a9 P 20193887ad994ccb8f3e9a4e4cad037c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 16:13:02.621850 5834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:02.620984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:40099 (local address 127.29.234.3:36099)
0924 16:13:02.621571 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:02.621681 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:40099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":268,"threads_started":1}
I20260924 16:13:02.622730 5834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:02.622382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:40099 (local address 127.29.234.3:40167)
0924 16:13:02.622514 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:02.622607 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:40099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 16:13:02.622761 5725 heartbeater.cc:503] Master 127.29.234.61:42395 was elected leader, sending a full tablet report...
I20260924 16:13:02.736394 5828 raft_consensus.cc:482] T fd2597f606e649749ac12252697762a9 P 76eabf034e914b95887d119fd3f8faa3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:13:02.736680 5828 raft_consensus.cc:504] T fd2597f606e649749ac12252697762a9 P 76eabf034e914b95887d119fd3f8faa3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20193887ad994ccb8f3e9a4e4cad037c" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 34375 } } peers { permanent_uuid: "14d33685ba3849c78f336923bf49e2d4" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 39763 } } peers { permanent_uuid: "76eabf034e914b95887d119fd3f8faa3" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 40063 } }
I20260924 16:13:02.738405 5828 leader_election.cc:288] T fd2597f606e649749ac12252697762a9 P 76eabf034e914b95887d119fd3f8faa3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20193887ad994ccb8f3e9a4e4cad037c (127.29.234.3:34375), 14d33685ba3849c78f336923bf49e2d4 (127.29.234.2:39763)
I20260924 16:13:02.739176 5833 raft_consensus.cc:482] T fd2597f606e649749ac12252697762a9 P 20193887ad994ccb8f3e9a4e4cad037c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:13:02.739414 5833 raft_consensus.cc:504] T fd2597f606e649749ac12252697762a9 P 20193887ad994ccb8f3e9a4e4cad037c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20193887ad994ccb8f3e9a4e4cad037c" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 34375 } } peers { permanent_uuid: "14d33685ba3849c78f336923bf49e2d4" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 39763 } } peers { permanent_uuid: "76eabf034e914b95887d119fd3f8faa3" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 40063 } }
I20260924 16:13:02.743222 5833 leader_election.cc:288] T fd2597f606e649749ac12252697762a9 P 20193887ad994ccb8f3e9a4e4cad037c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 14d33685ba3849c78f336923bf49e2d4 (127.29.234.2:39763), 76eabf034e914b95887d119fd3f8faa3 (127.29.234.1:40063)
I20260924 16:13:02.747432 5819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:02.738874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.1:34811 (local address 127.29.234.2:39763)
0924 16:13:02.739046 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:02.739054 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:02.739632 (+ 578us) server_negotiation.cc:407] Connection header received
0924 16:13:02.739746 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:02.739751 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:02.739838 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:02.739946 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:13:02.740485 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:02.741549 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:02.744918 (+ 3369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:02.746285 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:02.746450 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:02.746524 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:02.747049 (+ 525us) server_negotiation.cc:299] Negotiation successful
0924 16:13:02.747172 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 16:13:02.747473 5830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:02.738752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.2:39763 (local address 127.29.234.1:34811)
0924 16:13:02.739228 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:02.739244 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:02.739499 (+ 255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:02.739974 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:02.739982 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:02.740010 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:13:02.740295 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:02.740308 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:02.742108 (+ 1800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:02.742116 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:02.744736 (+ 2620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:02.744751 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:02.746665 (+ 1914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:02.746671 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:02.746858 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:02.746899 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 16:13:02.747011 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 16:13:02.747153 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"mutex_wait_us":68}
I20260924 16:13:02.748425 5835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:02.738898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.3:34375 (local address 127.29.234.1:53303)
0924 16:13:02.739707 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:02.739746 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:02.739945 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:02.740470 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:02.740474 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:02.740496 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:13:02.740753 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:02.740760 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:02.742261 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:02.742270 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:02.745489 (+ 3219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:02.745503 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:02.747590 (+ 2087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:02.747597 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:02.747773 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:02.747808 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 16:13:02.747977 (+ 169us) client_negotiation.cc:240] Negotiation successful
0924 16:13:02.748105 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":713,"thread_start_us":345,"threads_started":1}
I20260924 16:13:02.748449 5818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:02.738841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.1:53303 (local address 127.29.234.3:34375)
0924 16:13:02.739054 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:02.739059 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:02.739820 (+ 761us) server_negotiation.cc:407] Connection header received
0924 16:13:02.740130 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:02.740136 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:02.740214 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:02.740280 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:13:02.740899 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:02.742055 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:02.745652 (+ 3597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:02.747099 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:02.747258 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:02.747333 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:02.748033 (+ 700us) server_negotiation.cc:299] Negotiation successful
0924 16:13:02.748168 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":65}
I20260924 16:13:02.748350 5534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd2597f606e649749ac12252697762a9" candidate_uuid: "76eabf034e914b95887d119fd3f8faa3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14d33685ba3849c78f336923bf49e2d4" is_pre_election: true
I20260924 16:13:02.748792 5534 raft_consensus.cc:2454] T fd2597f606e649749ac12252697762a9 P 14d33685ba3849c78f336923bf49e2d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76eabf034e914b95887d119fd3f8faa3 in term 0.
I20260924 16:13:02.749054 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd2597f606e649749ac12252697762a9" candidate_uuid: "76eabf034e914b95887d119fd3f8faa3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20193887ad994ccb8f3e9a4e4cad037c" is_pre_election: true
I20260924 16:13:02.749305 5678 raft_consensus.cc:2454] T fd2597f606e649749ac12252697762a9 P 20193887ad994ccb8f3e9a4e4cad037c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76eabf034e914b95887d119fd3f8faa3 in term 0.
I20260924 16:13:02.749810 5316 leader_election.cc:302] T fd2597f606e649749ac12252697762a9 P 76eabf034e914b95887d119fd3f8faa3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14d33685ba3849c78f336923bf49e2d4, 76eabf034e914b95887d119fd3f8faa3; no voters:
I20260924 16:13:02.750252 5828 raft_consensus.cc:2781] T fd2597f606e649749ac12252697762a9 P 76eabf034e914b95887d119fd3f8faa3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:13:02.750388 5828 raft_consensus.cc:482] T fd2597f606e649749ac12252697762a9 P 76eabf034e914b95887d119fd3f8faa3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:13:02.750499 5828 raft_consensus.cc:3037] T fd2597f606e649749ac12252697762a9 P 76eabf034e914b95887d119fd3f8faa3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:13:02.751492 5834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:02.741498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:40063 (local address 127.29.234.3:43503)
0924 16:13:02.743500 (+ 2002us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:02.743521 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:02.743736 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:02.744272 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:02.744276 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:02.744302 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:13:02.744606 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:02.744617 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:02.746110 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:02.746115 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:02.748901 (+ 2786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:02.748917 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:02.750721 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:02.750728 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:02.750911 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:02.750947 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 16:13:02.751056 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 16:13:02.751184 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1865,"mutex_wait_us":4}
I20260924 16:13:02.751510 5821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:02.741819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.3:43503 (local address 127.29.234.1:40063)
0924 16:13:02.742026 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:02.742035 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:02.743604 (+ 1569us) server_negotiation.cc:407] Connection header received
0924 16:13:02.743927 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:02.743932 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:02.744006 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:02.744079 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:13:02.744781 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:02.745930 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:02.749070 (+ 3140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:02.750397 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:02.750544 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:02.750608 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:02.751103 (+ 495us) server_negotiation.cc:299] Negotiation successful
0924 16:13:02.751225 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":58}
I20260924 16:13:02.752140 5383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd2597f606e649749ac12252697762a9" candidate_uuid: "20193887ad994ccb8f3e9a4e4cad037c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76eabf034e914b95887d119fd3f8faa3" is_pre_election: true
I20260924 16:13:02.752532 5837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:02.741719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.3:41829 (local address 127.29.234.2:39763)
0924 16:13:02.743315 (+ 1596us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:02.743322 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:02.744039 (+ 717us) server_negotiation.cc:407] Connection header received
0924 16:13:02.744098 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:02.744102 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:02.744165 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:02.744247 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:13:02.745039 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:02.746245 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:02.750073 (+ 3828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:02.751313 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:02.751488 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:02.751583 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:02.752084 (+ 501us) server_negotiation.cc:299] Negotiation successful
0924 16:13:02.752208 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1469,"thread_start_us":1353,"threads_started":1}
I20260924 16:13:02.752560 5836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:02.741538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.2:39763 (local address 127.29.234.3:41829)
0924 16:13:02.743736 (+ 2198us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:02.743796 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:02.743937 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:02.744432 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:02.744436 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:02.744458 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:13:02.744831 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:02.744845 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:02.746397 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:02.746407 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:02.749892 (+ 3485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:02.749909 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:02.751739 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:02.751748 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:02.751929 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:02.751968 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 16:13:02.752056 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 16:13:02.752178 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2084,"mutex_wait_us":1210,"thread_start_us":1945,"threads_started":1}
I20260924 16:13:02.753147 5534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd2597f606e649749ac12252697762a9" candidate_uuid: "20193887ad994ccb8f3e9a4e4cad037c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14d33685ba3849c78f336923bf49e2d4" is_pre_election: true
I20260924 16:13:02.753391 5534 raft_consensus.cc:2454] T fd2597f606e649749ac12252697762a9 P 14d33685ba3849c78f336923bf49e2d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20193887ad994ccb8f3e9a4e4cad037c in term 0.
I20260924 16:13:02.753729 5828 raft_consensus.cc:504] T fd2597f606e649749ac12252697762a9 P 76eabf034e914b95887d119fd3f8faa3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20193887ad994ccb8f3e9a4e4cad037c" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 34375 } } peers { permanent_uuid: "14d33685ba3849c78f336923bf49e2d4" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 39763 } } peers { permanent_uuid: "76eabf034e914b95887d119fd3f8faa3" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 40063 } }
I20260924 16:13:02.753818 5610 leader_election.cc:302] T fd2597f606e649749ac12252697762a9 P 20193887ad994ccb8f3e9a4e4cad037c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14d33685ba3849c78f336923bf49e2d4, 20193887ad994ccb8f3e9a4e4cad037c; no voters:
I20260924 16:13:02.754081 5383 raft_consensus.cc:2379] T fd2597f606e649749ac12252697762a9 P 76eabf034e914b95887d119fd3f8faa3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20193887ad994ccb8f3e9a4e4cad037c in current term 1: Already voted for candidate 76eabf034e914b95887d119fd3f8faa3 in this term.
I20260924 16:13:02.754231 5833 raft_consensus.cc:2781] T fd2597f606e649749ac12252697762a9 P 20193887ad994ccb8f3e9a4e4cad037c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:13:02.754364 5833 raft_consensus.cc:482] T fd2597f606e649749ac12252697762a9 P 20193887ad994ccb8f3e9a4e4cad037c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:13:02.754472 5833 raft_consensus.cc:3037] T fd2597f606e649749ac12252697762a9 P 20193887ad994ccb8f3e9a4e4cad037c [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:13:02.754972 5828 leader_election.cc:288] T fd2597f606e649749ac12252697762a9 P 76eabf034e914b95887d119fd3f8faa3 [CANDIDATE]: Term 1 election: Requested vote from peers 20193887ad994ccb8f3e9a4e4cad037c (127.29.234.3:34375), 14d33685ba3849c78f336923bf49e2d4 (127.29.234.2:39763)
I20260924 16:13:02.755318 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd2597f606e649749ac12252697762a9" candidate_uuid: "76eabf034e914b95887d119fd3f8faa3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20193887ad994ccb8f3e9a4e4cad037c"
I20260924 16:13:02.755668 5534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd2597f606e649749ac12252697762a9" candidate_uuid: "76eabf034e914b95887d119fd3f8faa3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14d33685ba3849c78f336923bf49e2d4"
I20260924 16:13:02.755862 5534 raft_consensus.cc:3037] T fd2597f606e649749ac12252697762a9 P 14d33685ba3849c78f336923bf49e2d4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:13:02.756937 5833 raft_consensus.cc:504] T fd2597f606e649749ac12252697762a9 P 20193887ad994ccb8f3e9a4e4cad037c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20193887ad994ccb8f3e9a4e4cad037c" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 34375 } } peers { permanent_uuid: "14d33685ba3849c78f336923bf49e2d4" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 39763 } } peers { permanent_uuid: "76eabf034e914b95887d119fd3f8faa3" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 40063 } }
I20260924 16:13:02.757318 5678 raft_consensus.cc:2379] T fd2597f606e649749ac12252697762a9 P 20193887ad994ccb8f3e9a4e4cad037c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 76eabf034e914b95887d119fd3f8faa3 in current term 1: Already voted for candidate 20193887ad994ccb8f3e9a4e4cad037c in this term.
I20260924 16:13:02.757603 5833 leader_election.cc:288] T fd2597f606e649749ac12252697762a9 P 20193887ad994ccb8f3e9a4e4cad037c [CANDIDATE]: Term 1 election: Requested vote from peers 14d33685ba3849c78f336923bf49e2d4 (127.29.234.2:39763), 76eabf034e914b95887d119fd3f8faa3 (127.29.234.1:40063)
I20260924 16:13:02.758011 5533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd2597f606e649749ac12252697762a9" candidate_uuid: "20193887ad994ccb8f3e9a4e4cad037c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14d33685ba3849c78f336923bf49e2d4"
I20260924 16:13:02.758220 5383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd2597f606e649749ac12252697762a9" candidate_uuid: "20193887ad994ccb8f3e9a4e4cad037c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76eabf034e914b95887d119fd3f8faa3"
I20260924 16:13:02.758488 5383 raft_consensus.cc:2379] T fd2597f606e649749ac12252697762a9 P 76eabf034e914b95887d119fd3f8faa3 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 20193887ad994ccb8f3e9a4e4cad037c in current term 1: Already voted for candidate 76eabf034e914b95887d119fd3f8faa3 in this term.
I20260924 16:13:02.758569 5534 raft_consensus.cc:2454] T fd2597f606e649749ac12252697762a9 P 14d33685ba3849c78f336923bf49e2d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76eabf034e914b95887d119fd3f8faa3 in term 1.
I20260924 16:13:02.758980 5316 leader_election.cc:302] T fd2597f606e649749ac12252697762a9 P 76eabf034e914b95887d119fd3f8faa3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14d33685ba3849c78f336923bf49e2d4, 76eabf034e914b95887d119fd3f8faa3; no voters:
I20260924 16:13:02.759016 5610 leader_election.cc:302] T fd2597f606e649749ac12252697762a9 P 20193887ad994ccb8f3e9a4e4cad037c [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20193887ad994ccb8f3e9a4e4cad037c; no voters: 14d33685ba3849c78f336923bf49e2d4, 76eabf034e914b95887d119fd3f8faa3
I20260924 16:13:02.759248 5828 raft_consensus.cc:2781] T fd2597f606e649749ac12252697762a9 P 76eabf034e914b95887d119fd3f8faa3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:13:02.759395 5833 raft_consensus.cc:2726] T fd2597f606e649749ac12252697762a9 P 20193887ad994ccb8f3e9a4e4cad037c [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260924 16:13:02.759745 5828 raft_consensus.cc:686] T fd2597f606e649749ac12252697762a9 P 76eabf034e914b95887d119fd3f8faa3 [term 1 LEADER]: Becoming Leader. State: Replica: 76eabf034e914b95887d119fd3f8faa3, State: Running, Role: LEADER
I20260924 16:13:02.760077 5828 consensus_queue.cc:237] T fd2597f606e649749ac12252697762a9 P 76eabf034e914b95887d119fd3f8faa3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20193887ad994ccb8f3e9a4e4cad037c" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 34375 } } peers { permanent_uuid: "14d33685ba3849c78f336923bf49e2d4" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 39763 } } peers { permanent_uuid: "76eabf034e914b95887d119fd3f8faa3" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 40063 } }
I20260924 16:13:02.760985 5835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:02.760664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:40099 (local address 127.29.234.1:46049)
0924 16:13:02.760782 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:02.760861 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:40099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 16:13:02.763803 5055 catalog_manager.cc:5495] T fd2597f606e649749ac12252697762a9 P 76eabf034e914b95887d119fd3f8faa3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76eabf034e914b95887d119fd3f8faa3 (127.29.234.1). New cstate: current_term: 1 leader_uuid: "76eabf034e914b95887d119fd3f8faa3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20193887ad994ccb8f3e9a4e4cad037c" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 34375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14d33685ba3849c78f336923bf49e2d4" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 39763 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76eabf034e914b95887d119fd3f8faa3" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 40063 } health_report { overall_health: HEALTHY } } }
I20260924 16:13:02.822342 5821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:02.817700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45902 (local address 127.29.234.1:40063)
0924 16:13:02.817891 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:02.817895 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:02.817911 (+ 16us) server_negotiation.cc:407] Connection header received
0924 16:13:02.817999 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:02.818002 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:02.818075 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:02.818199 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:13:02.818514 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:02.819281 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:02.820463 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:02.820866 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:02.820953 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:02.821326 (+ 373us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:13:02.821882 (+ 556us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:13:02.821987 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:02.822063 (+ 76us) server_negotiation.cc:299] Negotiation successful
0924 16:13:02.822125 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 16:13:02.826306 5678 raft_consensus.cc:1261] T fd2597f606e649749ac12252697762a9 P 20193887ad994ccb8f3e9a4e4cad037c [term 1 FOLLOWER]: Refusing update from remote peer 76eabf034e914b95887d119fd3f8faa3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:13:02.826563 5534 raft_consensus.cc:1261] T fd2597f606e649749ac12252697762a9 P 14d33685ba3849c78f336923bf49e2d4 [term 1 FOLLOWER]: Refusing update from remote peer 76eabf034e914b95887d119fd3f8faa3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:13:02.827004 5838 consensus_queue.cc:1035] T fd2597f606e649749ac12252697762a9 P 76eabf034e914b95887d119fd3f8faa3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20193887ad994ccb8f3e9a4e4cad037c" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 34375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:13:02.827391 5828 consensus_queue.cc:1035] T fd2597f606e649749ac12252697762a9 P 76eabf034e914b95887d119fd3f8faa3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d33685ba3849c78f336923bf49e2d4" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 39763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:13:02.830132 5836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:02.829206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:40099 (local address 127.29.234.3:57333)
0924 16:13:02.829889 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:02.829996 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:40099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":570}
I20260924 16:13:02.831296 5831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:02.830481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:40099 (local address 127.29.234.2:39487)
0924 16:13:02.830832 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:02.831047 (+ 215us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:40099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204}
I20260924 16:13:02.842288 5842 mvcc.cc:205] Tried to move back new op lower bound from 7332931104049516544 to 7332931103789481984. Current Snapshot: MvccSnapshot[applied={T|T < 7332931104049516544}]
I20260924 16:13:02.843674 5831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:02.843120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:40099 (local address 127.29.234.2:47219)
0924 16:13:02.843347 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:02.843465 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:40099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260924 16:13:02.844962 5836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:02.844241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:40099 (local address 127.29.234.3:34849)
0924 16:13:02.844602 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:02.844742 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:40099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196}
I20260924 16:13:02.852121 5835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:02.851397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:40099 (local address 127.29.234.1:57261)
0924 16:13:02.851821 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:02.851946 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:40099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":299}
I20260924 16:13:02.857028 5835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:02.855397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:40099 (local address 127.29.234.1:44231)
0924 16:13:02.856757 (+ 1360us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:02.856848 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:40099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1263}
I20260924 16:13:02.865119 5057 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52150:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:13:02.865435 5057 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52150:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:13:02.871330 5057 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d44fd4dbe8224ecf93501b1d97a76304: Sending DeleteTablet for 3 replicas of tablet fd2597f606e649749ac12252697762a9
I20260924 16:13:02.872417 5657 tablet_service.cc:1514] Processing DeleteTablet for tablet fd2597f606e649749ac12252697762a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:13:02 UTC) from {username='oryx', principal='oryx/127.29.234.61@KRBTEST.COM'} at 127.0.0.1:48956
I20260924 16:13:02.872547 5513 tablet_service.cc:1514] Processing DeleteTablet for tablet fd2597f606e649749ac12252697762a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:13:02 UTC) from {username='oryx', principal='oryx/127.29.234.61@KRBTEST.COM'} at 127.0.0.1:40838
I20260924 16:13:02.873360 5362 tablet_service.cc:1514] Processing DeleteTablet for tablet fd2597f606e649749ac12252697762a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:13:02 UTC) from {username='oryx', principal='oryx/127.29.234.61@KRBTEST.COM'} at 127.0.0.1:45900
I20260924 16:13:02.875005 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 5286
I20260924 16:13:02.878537 5851 tablet_replica.cc:330] T fd2597f606e649749ac12252697762a9 P 20193887ad994ccb8f3e9a4e4cad037c: stopping tablet replica
I20260924 16:13:02.878866 5851 raft_consensus.cc:2229] T fd2597f606e649749ac12252697762a9 P 20193887ad994ccb8f3e9a4e4cad037c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:13:02.879349 5851 raft_consensus.cc:2258] T fd2597f606e649749ac12252697762a9 P 20193887ad994ccb8f3e9a4e4cad037c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:13:02.880546 5851 ts_tablet_manager.cc:1905] T fd2597f606e649749ac12252697762a9 P 20193887ad994ccb8f3e9a4e4cad037c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:13:02.885862 5853 tablet_replica.cc:330] T fd2597f606e649749ac12252697762a9 P 76eabf034e914b95887d119fd3f8faa3: stopping tablet replica
I20260924 16:13:02.887416 5852 tablet_replica.cc:330] T fd2597f606e649749ac12252697762a9 P 14d33685ba3849c78f336923bf49e2d4: stopping tablet replica
I20260924 16:13:02.887431 5851 ts_tablet_manager.cc:1918] T fd2597f606e649749ac12252697762a9 P 20193887ad994ccb8f3e9a4e4cad037c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:13:02.887679 5851 log.cc:1196] T fd2597f606e649749ac12252697762a9 P 20193887ad994ccb8f3e9a4e4cad037c: Deleting WAL directory at /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-2/wal/wals/fd2597f606e649749ac12252697762a9
I20260924 16:13:02.887610 5852 raft_consensus.cc:2229] T fd2597f606e649749ac12252697762a9 P 14d33685ba3849c78f336923bf49e2d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:13:02.886066 5853 raft_consensus.cc:2229] T fd2597f606e649749ac12252697762a9 P 76eabf034e914b95887d119fd3f8faa3 [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:13:02.887966 5852 raft_consensus.cc:2258] T fd2597f606e649749ac12252697762a9 P 14d33685ba3849c78f336923bf49e2d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:13:02.888309 5853 raft_consensus.cc:2258] T fd2597f606e649749ac12252697762a9 P 76eabf034e914b95887d119fd3f8faa3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:13:02.888870 5851 ts_tablet_manager.cc:1939] T fd2597f606e649749ac12252697762a9 P 20193887ad994ccb8f3e9a4e4cad037c: Deleting consensus metadata
I20260924 16:13:02.889246 5853 ts_tablet_manager.cc:1905] T fd2597f606e649749ac12252697762a9 P 76eabf034e914b95887d119fd3f8faa3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:13:02.890172 5042 catalog_manager.cc:4841] TS 20193887ad994ccb8f3e9a4e4cad037c (127.29.234.3:34375): tablet fd2597f606e649749ac12252697762a9 (table test-table [id=00b23a5660f048709da6c3e85b884d00]) successfully deleted
I20260924 16:13:02.891106 5852 ts_tablet_manager.cc:1905] T fd2597f606e649749ac12252697762a9 P 14d33685ba3849c78f336923bf49e2d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:13:02.898469 5852 ts_tablet_manager.cc:1918] T fd2597f606e649749ac12252697762a9 P 14d33685ba3849c78f336923bf49e2d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:13:02.898594 5852 log.cc:1196] T fd2597f606e649749ac12252697762a9 P 14d33685ba3849c78f336923bf49e2d4: Deleting WAL directory at /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-1/wal/wals/fd2597f606e649749ac12252697762a9
I20260924 16:13:02.899060 5852 ts_tablet_manager.cc:1939] T fd2597f606e649749ac12252697762a9 P 14d33685ba3849c78f336923bf49e2d4: Deleting consensus metadata
I20260924 16:13:02.900156 5041 catalog_manager.cc:4841] TS 14d33685ba3849c78f336923bf49e2d4 (127.29.234.2:39763): tablet fd2597f606e649749ac12252697762a9 (table test-table [id=00b23a5660f048709da6c3e85b884d00]) successfully deleted
I20260924 16:13:02.900832 5853 ts_tablet_manager.cc:1918] T fd2597f606e649749ac12252697762a9 P 76eabf034e914b95887d119fd3f8faa3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:13:02.900982 5853 log.cc:1196] T fd2597f606e649749ac12252697762a9 P 76eabf034e914b95887d119fd3f8faa3: Deleting WAL directory at /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790266323169505-30632-0/minicluster-data/ts-0/wal/wals/fd2597f606e649749ac12252697762a9
I20260924 16:13:02.901582 5853 ts_tablet_manager.cc:1939] T fd2597f606e649749ac12252697762a9 P 76eabf034e914b95887d119fd3f8faa3: Deleting consensus metadata
I20260924 16:13:02.902772 5043 catalog_manager.cc:4841] TS 76eabf034e914b95887d119fd3f8faa3 (127.29.234.1:40063): tablet fd2597f606e649749ac12252697762a9 (table test-table [id=00b23a5660f048709da6c3e85b884d00]) successfully deleted
I20260924 16:13:02.903620 5821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:02.876330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45904 (local address 127.29.234.1:40063)
0924 16:13:02.876560 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:02.876567 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:02.883766 (+ 7199us) server_negotiation.cc:407] Connection header received
0924 16:13:02.883975 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:02.883984 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:02.884116 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:02.884250 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:02.886160 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:02.887543 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:02.889861 (+ 2318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:02.890523 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:02.890660 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:02.896524 (+ 5864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:13:02.896549 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:13:02.896570 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:13:02.896612 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:13:02.900577 (+ 3965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:02.901412 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:02.901428 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:02.901432 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:02.901598 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:02.902037 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:02.902045 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:02.902048 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:02.902457 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:13:02.902841 (+ 384us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:02.903137 (+ 296us) server_negotiation.cc:299] Negotiation successful
0924 16:13:02.903301 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 16:13:02.904230 5423 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:13:03.008870 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 5286
I20260924 16:13:03.029158 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 5441
I20260924 16:13:03.042634 5837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:03.030006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40854 (local address 127.29.234.2:39763)
0924 16:13:03.030203 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:03.030209 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:03.030282 (+ 73us) server_negotiation.cc:407] Connection header received
0924 16:13:03.030455 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:03.030459 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:03.030525 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:03.030612 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:03.032001 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:03.032784 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:03.034110 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:03.034542 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:03.034638 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:03.037149 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:13:03.037165 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:13:03.037186 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:13:03.037225 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:13:03.040169 (+ 2944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:03.040832 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:03.040839 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:03.040842 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:03.040972 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:03.041280 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:03.041283 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:03.041285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:03.041698 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:13:03.041938 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:03.042141 (+ 203us) server_negotiation.cc:299] Negotiation successful
0924 16:13:03.042307 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 16:13:03.043362 5575 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:13:03.045459 5802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:03.044876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:40099 (local address 127.0.0.1:38026)
0924 16:13:03.045083 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:03.045199 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:40099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260924 16:13:03.046772 5802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:03.046238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:40099 (local address 127.0.0.1:38042)
0924 16:13:03.046412 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:03.046529 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:40099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 16:13:03.166888 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 5441
I20260924 16:13:03.190367 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 5593
I20260924 16:13:03.203029 5818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:03.191116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48970 (local address 127.29.234.3:34375)
0924 16:13:03.191348 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:03.191357 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:03.191383 (+ 26us) server_negotiation.cc:407] Connection header received
0924 16:13:03.191510 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:03.191514 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:03.191578 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:03.191667 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:03.193037 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:03.193874 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:03.195022 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:03.195414 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:03.195504 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:03.198186 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:13:03.198200 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:13:03.198218 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:13:03.198250 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:13:03.200715 (+ 2465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:03.201280 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:03.201287 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:03.201289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:03.201465 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:03.201757 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:03.201760 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:03.201762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:03.202097 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:13:03.202299 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:03.202551 (+ 252us) server_negotiation.cc:299] Negotiation successful
0924 16:13:03.202733 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260924 16:13:03.203707 5718 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:13:03.300235 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 5593
I20260924 16:13:03.317972 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 4914
I20260924 16:13:03.319039 5087 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:13:03.443291 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 4914
I20260924 16:13:03.464977 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 5011
I20260924 16:13:03.466121 5278 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:13:03.585152 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 5011
2026-09-24T16:13:03Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (10037 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:13:03 dist-test-slave-v5j9 krb5kdc[5862](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:13:03 dist-test-slave-v5j9 krb5kdc[5862](info): set up 2 sockets
Sep 24 16:13:03 dist-test-slave-v5j9 krb5kdc[5862](info): commencing operation
krb5kdc: starting...
W20260924 16:13:04.701864 30632 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.002s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Sep 24 16:13:04 dist-test-slave-v5j9 krb5kdc[5862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:13:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:13:04Z Disabled control of system clock
WARNING: no policy specified for oryx/127.29.234.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.234.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.234.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/krb5kdc/oryx_127.29.234.62.keytab.
Entry for principal oryx/127.29.234.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/krb5kdc/oryx_127.29.234.62.keytab.
I20260924 16:13:04.929692 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:40365
--webserver_interface=127.29.234.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/krb5kdc/oryx_127.29.234.62.keytab
--principal=oryx/127.29.234.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:38831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.234.62:40365
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:13:05.262462 5875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:13:05.262773 5875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:13:05.262883 5875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:13:05.270197 5875 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:13:05.270308 5875 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 16:13:05.270350 5875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:13:05.270411 5875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:13:05.270479 5875 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:13:05.270536 5875 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:13:05.280864 5875 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:38831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.234.62:40365
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/krb5kdc/oryx_127.29.234.62.keytab
--principal=oryx/127.29.234.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:40365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.234.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:13:05.282469 5875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:13:05.283854 5875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:13:05.296314 5880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:13:05.296257 5881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:13:05.296288 5883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:13:05.297080 5875 server_base.cc:1034] running on GCE node
I20260924 16:13:05.297974 5875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:13:05.299436 5875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:13:05.300725 5875 hybrid_clock.cc:648] HybridClock initialized: now 1790266385300660 us; error 45 us; skew 500 ppm
I20260924 16:13:05.301039 5875 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:13:05 dist-test-slave-v5j9 krb5kdc[5862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266385, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.234.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:13:05.304332 5875 init.cc:377] Logged in from keytab as oryx/127.29.234.62@KRBTEST.COM (short username oryx)
I20260924 16:13:05.305971 5875 webserver.cc:460] Webserver started at http://127.29.234.62:39699/ using document root <none> and password file <none>
I20260924 16:13:05.306732 5875 fs_manager.cc:359] Metadata directory not provided
I20260924 16:13:05.306859 5875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:13:05.307209 5875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:13:05.310374 5875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/minicluster-data/master-0/data/instance:
uuid: "441441046a974081ab6e0e4a1576c8f7"
format_stamp: "Formatted at 2026-09-24 16:13:05 on dist-test-slave-v5j9"
server_key: "5a6fa9f86ae42ea1c36ca655be9fd023"
server_key_iv: "fa58b28ba46f7fae5bc0b32ffce760a0"
server_key_version: "encryptionkey@0"
I20260924 16:13:05.311183 5875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/minicluster-data/master-0/wal/instance:
uuid: "441441046a974081ab6e0e4a1576c8f7"
format_stamp: "Formatted at 2026-09-24 16:13:05 on dist-test-slave-v5j9"
server_key: "5a6fa9f86ae42ea1c36ca655be9fd023"
server_key_iv: "fa58b28ba46f7fae5bc0b32ffce760a0"
server_key_version: "encryptionkey@0"
I20260924 16:13:05.316460 5875 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 16:13:05.320048 5890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:13:05.321441 5875 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 16:13:05.321615 5875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/minicluster-data/master-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/minicluster-data/master-0/wal
uuid: "441441046a974081ab6e0e4a1576c8f7"
format_stamp: "Formatted at 2026-09-24 16:13:05 on dist-test-slave-v5j9"
server_key: "5a6fa9f86ae42ea1c36ca655be9fd023"
server_key_iv: "fa58b28ba46f7fae5bc0b32ffce760a0"
server_key_version: "encryptionkey@0"
I20260924 16:13:05.321838 5875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:13:05.345247 5875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:13:05.349401 5875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:13:05.349678 5875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:13:05.371925 5875 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.62:40365
I20260924 16:13:05.371848 5954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.62:40365 every 8 connection(s)
I20260924 16:13:05.373235 5875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
I20260924 16:13:05.377074 5955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:13:05.381311 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 5875
I20260924 16:13:05.381544 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/minicluster-data/master-0/wal/instance
I20260924 16:13:05.382444 30632 external_mini_cluster.cc:1428] Setting key 704583d240ce048be9468c7f94b5fa09
I20260924 16:13:05.389596 5955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 441441046a974081ab6e0e4a1576c8f7: Bootstrap starting.
I20260924 16:13:05.393180 5955 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 441441046a974081ab6e0e4a1576c8f7: Neither blocks nor log segments found. Creating new log.
I20260924 16:13:05.394578 5955 log.cc:824] T 00000000000000000000000000000000 P 441441046a974081ab6e0e4a1576c8f7: Log is configured to *not* fsync() on all Append() calls
I20260924 16:13:05.397603 5955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 441441046a974081ab6e0e4a1576c8f7: No bootstrap required, opened a new log
Sep 24 16:13:05 dist-test-slave-v5j9 krb5kdc[5862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM
I20260924 16:13:05.406471 5958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:05.384545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49432 (local address 127.29.234.62:40365)
0924 16:13:05.385335 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:05.385390 (+ 55us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:05.385444 (+ 54us) server_negotiation.cc:407] Connection header received
0924 16:13:05.387940 (+ 2496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:05.387963 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:05.388342 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:05.388644 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:05.390449 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:05.391746 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:05.393497 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:05.394607 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:05.394795 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:05.399163 (+ 4368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:13:05.399171 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:13:05.399186 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:13:05.399223 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:13:05.402403 (+ 3180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:05.403213 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:05.403218 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:05.403224 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:05.403400 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:05.403799 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:05.403802 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:05.403804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:05.404315 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:13:05.404541 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:05.404962 (+ 421us) server_negotiation.cc:299] Negotiation successful
0924 16:13:05.405400 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":371,"threads_started":1}
I20260924 16:13:05.407114 5955 raft_consensus.cc:348] T 00000000000000000000000000000000 P 441441046a974081ab6e0e4a1576c8f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "441441046a974081ab6e0e4a1576c8f7" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 40365 } }
I20260924 16:13:05.407485 5955 raft_consensus.cc:374] T 00000000000000000000000000000000 P 441441046a974081ab6e0e4a1576c8f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:13:05.407604 5955 raft_consensus.cc:729] T 00000000000000000000000000000000 P 441441046a974081ab6e0e4a1576c8f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 441441046a974081ab6e0e4a1576c8f7, State: Initialized, Role: FOLLOWER
I20260924 16:13:05.408274 5955 consensus_queue.cc:260] T 00000000000000000000000000000000 P 441441046a974081ab6e0e4a1576c8f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "441441046a974081ab6e0e4a1576c8f7" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 40365 } }
I20260924 16:13:05.408552 5955 raft_consensus.cc:388] T 00000000000000000000000000000000 P 441441046a974081ab6e0e4a1576c8f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:13:05.408702 5955 raft_consensus.cc:482] T 00000000000000000000000000000000 P 441441046a974081ab6e0e4a1576c8f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:13:05.408877 5955 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 441441046a974081ab6e0e4a1576c8f7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:13:05.411126 5955 raft_consensus.cc:504] T 00000000000000000000000000000000 P 441441046a974081ab6e0e4a1576c8f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "441441046a974081ab6e0e4a1576c8f7" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 40365 } }
I20260924 16:13:05.411573 5955 leader_election.cc:302] T 00000000000000000000000000000000 P 441441046a974081ab6e0e4a1576c8f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 441441046a974081ab6e0e4a1576c8f7; no voters:
I20260924 16:13:05.412122 5955 leader_election.cc:288] T 00000000000000000000000000000000 P 441441046a974081ab6e0e4a1576c8f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:13:05.412236 5960 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 441441046a974081ab6e0e4a1576c8f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:13:05.413405 5960 raft_consensus.cc:686] T 00000000000000000000000000000000 P 441441046a974081ab6e0e4a1576c8f7 [term 1 LEADER]: Becoming Leader. State: Replica: 441441046a974081ab6e0e4a1576c8f7, State: Running, Role: LEADER
I20260924 16:13:05.413918 5960 consensus_queue.cc:237] T 00000000000000000000000000000000 P 441441046a974081ab6e0e4a1576c8f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "441441046a974081ab6e0e4a1576c8f7" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 40365 } }
I20260924 16:13:05.414529 5955 sys_catalog.cc:564] T 00000000000000000000000000000000 P 441441046a974081ab6e0e4a1576c8f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:13:05.416720 5962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 441441046a974081ab6e0e4a1576c8f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 441441046a974081ab6e0e4a1576c8f7. Latest consensus state: current_term: 1 leader_uuid: "441441046a974081ab6e0e4a1576c8f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "441441046a974081ab6e0e4a1576c8f7" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 40365 } } }
I20260924 16:13:05.416709 5961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 441441046a974081ab6e0e4a1576c8f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "441441046a974081ab6e0e4a1576c8f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "441441046a974081ab6e0e4a1576c8f7" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 40365 } } }
I20260924 16:13:05.417133 5962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 441441046a974081ab6e0e4a1576c8f7 [sys.catalog]: This master's current role is: LEADER
I20260924 16:13:05.417273 5961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 441441046a974081ab6e0e4a1576c8f7 [sys.catalog]: This master's current role is: LEADER
I20260924 16:13:05.421964 5965 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:13:05.426985 5965 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:13:05.436442 5965 catalog_manager.cc:1347] Generated new cluster ID: 2390ba3c9d984d0e9f19d7df0e744392
I20260924 16:13:05.436558 5965 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:13:05.492018 5965 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:13:05.493983 5965 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:13:05.507887 5965 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 441441046a974081ab6e0e4a1576c8f7: Generated new TSK 0
I20260924 16:13:05.508960 5965 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.29.234.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.234.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.234.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/krb5kdc/oryx_127.29.234.1.keytab.
Entry for principal oryx/127.29.234.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/krb5kdc/oryx_127.29.234.1.keytab.
I20260924 16:13:05.567245 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.1:0
--local_ip_for_outbound_sockets=127.29.234.1
--webserver_interface=127.29.234.1
--webserver_port=0
--tserver_master_addrs=127.29.234.62:40365
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/krb5kdc/oryx_127.29.234.1.keytab
--principal=oryx/127.29.234.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:38831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:13:05.908818 5981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:13:05.909197 5981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:13:05.909421 5981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:13:05.916853 5981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:13:05.917008 5981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:13:05.917152 5981 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 16:13:05.917296 5981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.1
I20260924 16:13:05.927593 5981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:38831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/krb5kdc/oryx_127.29.234.1.keytab
--principal=oryx/127.29.234.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.234.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.234.62:40365
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.1
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:13:05.928941 5981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:13:05.930351 5981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:13:05.943476 5989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:13:05.943436 5987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:13:05.943514 5986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:13:05.944356 5981 server_base.cc:1034] running on GCE node
I20260924 16:13:05.945039 5981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:13:05.946058 5981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:13:05.947269 5981 hybrid_clock.cc:648] HybridClock initialized: now 1790266385947208 us; error 33 us; skew 500 ppm
I20260924 16:13:05.947584 5981 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:13:05 dist-test-slave-v5j9 krb5kdc[5862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266385, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.234.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:13:05.950861 5981 init.cc:377] Logged in from keytab as oryx/127.29.234.1@KRBTEST.COM (short username oryx)
I20260924 16:13:05.952193 5981 webserver.cc:460] Webserver started at http://127.29.234.1:40117/ using document root <none> and password file <none>
I20260924 16:13:05.952929 5981 fs_manager.cc:359] Metadata directory not provided
I20260924 16:13:05.953042 5981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:13:05.953437 5981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:13:05.956326 5981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/minicluster-data/ts-0/data/instance:
uuid: "284ce6491d364d2a83dad33520490442"
format_stamp: "Formatted at 2026-09-24 16:13:05 on dist-test-slave-v5j9"
server_key: "891e94efdf52f50b9a1786955f8f1c6a"
server_key_iv: "1fcc7c14863470cb41fc5696578d43eb"
server_key_version: "encryptionkey@0"
I20260924 16:13:05.957135 5981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/minicluster-data/ts-0/wal/instance:
uuid: "284ce6491d364d2a83dad33520490442"
format_stamp: "Formatted at 2026-09-24 16:13:05 on dist-test-slave-v5j9"
server_key: "891e94efdf52f50b9a1786955f8f1c6a"
server_key_iv: "1fcc7c14863470cb41fc5696578d43eb"
server_key_version: "encryptionkey@0"
I20260924 16:13:05.962420 5981 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260924 16:13:05.965940 5996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:13:05.967248 5981 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260924 16:13:05.967425 5981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/minicluster-data/ts-0/wal
uuid: "284ce6491d364d2a83dad33520490442"
format_stamp: "Formatted at 2026-09-24 16:13:05 on dist-test-slave-v5j9"
server_key: "891e94efdf52f50b9a1786955f8f1c6a"
server_key_iv: "1fcc7c14863470cb41fc5696578d43eb"
server_key_version: "encryptionkey@0"
I20260924 16:13:05.967658 5981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:13:05.989483 5981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:13:05.993649 5981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:13:05.993959 5981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:13:05.996693 5981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:13:05.996824 5981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:13:05.996997 5981 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:13:05.997076 5981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 16:13:06 dist-test-slave-v5j9 krb5kdc[5862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266385, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.234.1@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM
I20260924 16:13:06.021119 6005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:05.997494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.1:36569 (local address 127.29.234.62:40365)
0924 16:13:05.998089 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:05.998096 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:05.999326 (+ 1230us) server_negotiation.cc:407] Connection header received
0924 16:13:06.001502 (+ 2176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:06.001508 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:06.001608 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:06.001726 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:06.003938 (+ 2212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:06.004984 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:06.007973 (+ 2989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:06.008706 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:06.008845 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:06.013044 (+ 4199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:13:06.013050 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:13:06.013055 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:13:06.013097 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:13:06.016966 (+ 3869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:06.018061 (+ 1095us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:06.018068 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:06.018072 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:06.018230 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:06.018931 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:06.018937 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:06.018940 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:06.019268 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:13:06.019442 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:06.020509 (+ 1067us) server_negotiation.cc:299] Negotiation successful
0924 16:13:06.020721 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":348,"threads_started":1}
I20260924 16:13:06.022249 6006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:05.998195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:40365 (local address 127.29.234.1:36569)
0924 16:13:05.999138 (+ 943us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:05.999181 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:06.001196 (+ 2015us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:06.001991 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:06.001999 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:06.002834 (+ 835us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:13:06.003748 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:06.003768 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:06.005169 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:06.005176 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:06.007600 (+ 2424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:06.007623 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:06.008981 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:06.008991 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:06.009211 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:06.010015 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:13:06.010069 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:13:06.012799 (+ 2730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:13:06.017147 (+ 4348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:13:06.017158 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:13:06.017178 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:13:06.017928 (+ 750us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:13:06.018371 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:13:06.018379 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:13:06.018384 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:13:06.018767 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:13:06.019536 (+ 769us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:13:06.019544 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:13:06.020067 (+ 523us) client_negotiation.cc:769] Sending connection context
0924 16:13:06.020426 (+ 359us) client_negotiation.cc:240] Negotiation successful
0924 16:13:06.020987 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"thread_start_us":415,"threads_started":1}
I20260924 16:13:06.043830 6005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:06.028628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.1:53761 (local address 127.29.234.62:40365)
0924 16:13:06.028841 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:06.028859 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:06.029509 (+ 650us) server_negotiation.cc:407] Connection header received
0924 16:13:06.029746 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:06.029753 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:06.029819 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:06.029894 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:06.031335 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:06.032003 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:06.034641 (+ 2638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:06.035182 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:06.035315 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:06.037777 (+ 2462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:13:06.037783 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:13:06.037788 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:13:06.037826 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:13:06.040963 (+ 3137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:06.041876 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:06.041881 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:06.041885 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:06.042009 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:06.042379 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:06.042384 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:06.042387 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:06.042641 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:13:06.042790 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:06.043269 (+ 479us) server_negotiation.cc:299] Negotiation successful
0924 16:13:06.043479 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 16:13:06.043937 6067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:06.028606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:40365 (local address 127.29.234.1:53761)
0924 16:13:06.029338 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:06.029391 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:06.029596 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:06.029966 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:06.029974 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:06.030691 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:13:06.031185 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:06.031201 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:06.032150 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:06.032158 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:06.034451 (+ 2293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:06.034472 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:06.035441 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:06.035450 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:06.035722 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:06.036474 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:13:06.036502 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:13:06.037595 (+ 1093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:13:06.041122 (+ 3527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:13:06.041126 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:13:06.041128 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:13:06.041746 (+ 618us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:13:06.042123 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:13:06.042128 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:13:06.042131 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:13:06.042260 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:13:06.042761 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:13:06.042765 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:13:06.042991 (+ 226us) client_negotiation.cc:769] Sending connection context
0924 16:13:06.043154 (+ 163us) client_negotiation.cc:240] Negotiation successful
0924 16:13:06.043474 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":374,"threads_started":1}
I20260924 16:13:06.052392 5981 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.1:41013
I20260924 16:13:06.052421 6118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.1:41013 every 8 connection(s)
I20260924 16:13:06.053715 5981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
I20260924 16:13:06.055696 6119 heartbeater.cc:347] Connected to a master server at 127.29.234.62:40365
I20260924 16:13:06.055939 6119 heartbeater.cc:464] Registering TS with master...
I20260924 16:13:06.056571 6119 heartbeater.cc:511] Master 127.29.234.62:40365 requested a full tablet report, sending...
I20260924 16:13:06.058524 5908 ts_manager.cc:194] Registered new tserver with Master: 284ce6491d364d2a83dad33520490442 (127.29.234.1:41013)
I20260924 16:13:06.059970 5908 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.234.1@KRBTEST.COM'} at 127.29.234.1:36569
I20260924 16:13:06.063611 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 5981
I20260924 16:13:06.063818 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/minicluster-data/ts-0/wal/instance
I20260924 16:13:06.064481 30632 external_mini_cluster.cc:1428] Setting key a334bec5f578df21b03dacbf75a53640
WARNING: no policy specified for oryx/127.29.234.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.234.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.234.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/krb5kdc/oryx_127.29.234.2.keytab.
Entry for principal oryx/127.29.234.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/krb5kdc/oryx_127.29.234.2.keytab.
I20260924 16:13:06.112353 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.2:0
--local_ip_for_outbound_sockets=127.29.234.2
--webserver_interface=127.29.234.2
--webserver_port=0
--tserver_master_addrs=127.29.234.62:40365
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/krb5kdc/oryx_127.29.234.2.keytab
--principal=oryx/127.29.234.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:38831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:13:06.438112 6123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:13:06.438390 6123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:13:06.438560 6123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:13:06.445782 6123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:13:06.445889 6123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:13:06.446029 6123 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 16:13:06.446182 6123 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.2
I20260924 16:13:06.456063 6123 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:38831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/krb5kdc/oryx_127.29.234.2.keytab
--principal=oryx/127.29.234.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.234.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.234.62:40365
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.2
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:13:06.457453 6123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:13:06.458858 6123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:13:06.471947 6128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:13:06.471967 6131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:13:06.471866 6129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:13:06.472818 6123 server_base.cc:1034] running on GCE node
I20260924 16:13:06.473440 6123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:13:06.474404 6123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:13:06.475617 6123 hybrid_clock.cc:648] HybridClock initialized: now 1790266386475563 us; error 35 us; skew 500 ppm
I20260924 16:13:06.475875 6123 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:13:06 dist-test-slave-v5j9 krb5kdc[5862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266386, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.234.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:13:06.478861 6123 init.cc:377] Logged in from keytab as oryx/127.29.234.2@KRBTEST.COM (short username oryx)
I20260924 16:13:06.480134 6123 webserver.cc:460] Webserver started at http://127.29.234.2:45813/ using document root <none> and password file <none>
I20260924 16:13:06.480777 6123 fs_manager.cc:359] Metadata directory not provided
I20260924 16:13:06.480863 6123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:13:06.481125 6123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:13:06.483983 6123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/minicluster-data/ts-1/data/instance:
uuid: "43317d13dde547deabe618d1ea6110fa"
format_stamp: "Formatted at 2026-09-24 16:13:06 on dist-test-slave-v5j9"
server_key: "93d994e1733b21406d8dcfd0f33a35ea"
server_key_iv: "2bf895f99b325c63a48af2cb8dd68e5a"
server_key_version: "encryptionkey@0"
I20260924 16:13:06.484714 6123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/minicluster-data/ts-1/wal/instance:
uuid: "43317d13dde547deabe618d1ea6110fa"
format_stamp: "Formatted at 2026-09-24 16:13:06 on dist-test-slave-v5j9"
server_key: "93d994e1733b21406d8dcfd0f33a35ea"
server_key_iv: "2bf895f99b325c63a48af2cb8dd68e5a"
server_key_version: "encryptionkey@0"
I20260924 16:13:06.489584 6123 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260924 16:13:06.492861 6138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:13:06.494197 6123 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:13:06.494320 6123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/minicluster-data/ts-1/wal
uuid: "43317d13dde547deabe618d1ea6110fa"
format_stamp: "Formatted at 2026-09-24 16:13:06 on dist-test-slave-v5j9"
server_key: "93d994e1733b21406d8dcfd0f33a35ea"
server_key_iv: "2bf895f99b325c63a48af2cb8dd68e5a"
server_key_version: "encryptionkey@0"
I20260924 16:13:06.494500 6123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:13:06.517871 6123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:13:06.521997 6123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:13:06.522229 6123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:13:06.525175 6123 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:13:06.525292 6123 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:13:06.525456 6123 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:13:06.525552 6123 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 16:13:06 dist-test-slave-v5j9 krb5kdc[5862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266386, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.234.2@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM
I20260924 16:13:06.550964 6005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:06.525710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.2:46231 (local address 127.29.234.62:40365)
0924 16:13:06.525972 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:06.525979 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:06.527585 (+ 1606us) server_negotiation.cc:407] Connection header received
0924 16:13:06.530692 (+ 3107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:06.530701 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:06.530796 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:06.530940 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:06.533271 (+ 2331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:06.534363 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:06.536922 (+ 2559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:06.537582 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:06.537725 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:06.542720 (+ 4995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:13:06.542728 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:13:06.542734 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:13:06.542777 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:13:06.546345 (+ 3568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:06.547360 (+ 1015us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:06.547367 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:06.547373 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:06.547571 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:06.548208 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:06.548216 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:06.548221 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:06.548572 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:13:06.548747 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:06.550420 (+ 1673us) server_negotiation.cc:299] Negotiation successful
0924 16:13:06.550640 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260924 16:13:06.553403 6147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:06.526421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:40365 (local address 127.29.234.2:46231)
0924 16:13:06.527382 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:06.527444 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:06.530237 (+ 2793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:06.531211 (+ 974us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:06.531219 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:06.532044 (+ 825us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:13:06.533047 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:06.533080 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:06.534549 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:06.534553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:06.536749 (+ 2196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:06.536768 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:06.537862 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:06.537870 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:06.538150 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:06.538987 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:13:06.539021 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:13:06.542389 (+ 3368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:13:06.546495 (+ 4106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:13:06.546505 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:13:06.546526 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:13:06.547218 (+ 692us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:13:06.547737 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:13:06.547744 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:13:06.547751 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:13:06.548018 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:13:06.548851 (+ 833us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:13:06.548861 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:13:06.549440 (+ 579us) client_negotiation.cc:769] Sending connection context
0924 16:13:06.549868 (+ 428us) client_negotiation.cc:240] Negotiation successful
0924 16:13:06.551486 (+ 1618us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":374,"threads_started":1}
I20260924 16:13:06.572695 6123 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.2:36571
I20260924 16:13:06.572719 6259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.2:36571 every 8 connection(s)
I20260924 16:13:06.574146 6123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
I20260924 16:13:06.575485 6005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:06.559751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.2:33795 (local address 127.29.234.62:40365)
0924 16:13:06.559994 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:06.560004 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:06.560575 (+ 571us) server_negotiation.cc:407] Connection header received
0924 16:13:06.560908 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:06.560916 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:06.561035 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:06.561110 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:06.562678 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:06.563611 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:06.565847 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:06.566350 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:06.566497 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:06.569006 (+ 2509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:13:06.569014 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:13:06.569022 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:13:06.569068 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:13:06.572402 (+ 3334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:06.573204 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:06.573210 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:06.573214 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:06.573425 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:06.573821 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:06.573826 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:06.573829 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:06.574319 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:13:06.574470 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:06.574912 (+ 442us) server_negotiation.cc:299] Negotiation successful
0924 16:13:06.575126 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 16:13:06.575634 6229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:06.559702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:40365 (local address 127.29.234.2:33795)
0924 16:13:06.560458 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:06.560488 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:06.560723 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:06.561393 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:06.561401 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:06.562102 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:13:06.562515 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:06.562533 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:06.563748 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:06.563765 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:06.565690 (+ 1925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:06.565710 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:06.566597 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:06.566606 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:06.566862 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:06.567651 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:13:06.567681 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:13:06.568780 (+ 1099us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:13:06.572538 (+ 3758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:13:06.572547 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:13:06.572551 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:13:06.573084 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:13:06.573547 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:13:06.573552 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:13:06.573555 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:13:06.573687 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:13:06.574433 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:13:06.574438 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:13:06.574636 (+ 198us) client_negotiation.cc:769] Sending connection context
0924 16:13:06.574811 (+ 175us) client_negotiation.cc:240] Negotiation successful
0924 16:13:06.575148 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":360,"threads_started":1}
I20260924 16:13:06.576176 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 6123
I20260924 16:13:06.576332 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/minicluster-data/ts-1/wal/instance
I20260924 16:13:06.576788 6260 heartbeater.cc:347] Connected to a master server at 127.29.234.62:40365
I20260924 16:13:06.576874 30632 external_mini_cluster.cc:1428] Setting key b9f3becb59110b6a47a7e5fad9101fc0
I20260924 16:13:06.577090 6260 heartbeater.cc:464] Registering TS with master...
I20260924 16:13:06.578147 6260 heartbeater.cc:511] Master 127.29.234.62:40365 requested a full tablet report, sending...
I20260924 16:13:06.580124 5908 ts_manager.cc:194] Registered new tserver with Master: 43317d13dde547deabe618d1ea6110fa (127.29.234.2:36571)
I20260924 16:13:06.580973 5908 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.234.2@KRBTEST.COM'} at 127.29.234.2:46231
WARNING: no policy specified for oryx/127.29.234.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.234.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.234.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/krb5kdc/oryx_127.29.234.3.keytab.
Entry for principal oryx/127.29.234.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/krb5kdc/oryx_127.29.234.3.keytab.
I20260924 16:13:06.625587 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.3:0
--local_ip_for_outbound_sockets=127.29.234.3
--webserver_interface=127.29.234.3
--webserver_port=0
--tserver_master_addrs=127.29.234.62:40365
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/krb5kdc/oryx_127.29.234.3.keytab
--principal=oryx/127.29.234.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:38831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:13:06.954025 6264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:13:06.954330 6264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:13:06.954493 6264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:13:06.962029 6264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:13:06.962221 6264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:13:06.962376 6264 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 16:13:06.962528 6264 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.3
I20260924 16:13:06.972496 6264 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:38831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/krb5kdc/oryx_127.29.234.3.keytab
--principal=oryx/127.29.234.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.234.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.234.62:40365
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.3
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:13:06.973861 6264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:13:06.975368 6264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:13:06.989203 6269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:13:06.989115 6272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:13:06.989143 6270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:13:06.989976 6264 server_base.cc:1034] running on GCE node
I20260924 16:13:06.990662 6264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:13:06.991621 6264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:13:06.992874 6264 hybrid_clock.cc:648] HybridClock initialized: now 1790266386992811 us; error 44 us; skew 500 ppm
I20260924 16:13:06.993201 6264 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:13:06 dist-test-slave-v5j9 krb5kdc[5862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266386, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.234.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:13:06.996645 6264 init.cc:377] Logged in from keytab as oryx/127.29.234.3@KRBTEST.COM (short username oryx)
I20260924 16:13:06.998215 6264 webserver.cc:460] Webserver started at http://127.29.234.3:41387/ using document root <none> and password file <none>
I20260924 16:13:06.999043 6264 fs_manager.cc:359] Metadata directory not provided
I20260924 16:13:06.999171 6264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:13:06.999501 6264 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:13:07.002566 6264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/minicluster-data/ts-2/data/instance:
uuid: "a204f0e9eec340e1bd4d74175dd392a2"
format_stamp: "Formatted at 2026-09-24 16:13:06 on dist-test-slave-v5j9"
server_key: "e7b0c88de9c6702b3c6e59b6b5d4530b"
server_key_iv: "872695354f488353b16cbed9f222c096"
server_key_version: "encryptionkey@0"
I20260924 16:13:07.003341 6264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/minicluster-data/ts-2/wal/instance:
uuid: "a204f0e9eec340e1bd4d74175dd392a2"
format_stamp: "Formatted at 2026-09-24 16:13:06 on dist-test-slave-v5j9"
server_key: "e7b0c88de9c6702b3c6e59b6b5d4530b"
server_key_iv: "872695354f488353b16cbed9f222c096"
server_key_version: "encryptionkey@0"
I20260924 16:13:07.008687 6264 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260924 16:13:07.012243 6280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:13:07.013659 6264 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 16:13:07.013831 6264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/minicluster-data/ts-2/wal
uuid: "a204f0e9eec340e1bd4d74175dd392a2"
format_stamp: "Formatted at 2026-09-24 16:13:06 on dist-test-slave-v5j9"
server_key: "e7b0c88de9c6702b3c6e59b6b5d4530b"
server_key_iv: "872695354f488353b16cbed9f222c096"
server_key_version: "encryptionkey@0"
I20260924 16:13:07.014062 6264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:13:07.033154 6264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:13:07.037246 6264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:13:07.037547 6264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:13:07.040062 6264 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:13:07.040171 6264 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:13:07.040335 6264 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:13:07.040416 6264 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 16:13:07 dist-test-slave-v5j9 krb5kdc[5862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266386, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.234.3@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM
I20260924 16:13:07.062726 6119 heartbeater.cc:503] Master 127.29.234.62:40365 was elected leader, sending a full tablet report...
I20260924 16:13:07.065500 6005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:07.040953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.3:36565 (local address 127.29.234.62:40365)
0924 16:13:07.041142 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:07.041149 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:07.042636 (+ 1487us) server_negotiation.cc:407] Connection header received
0924 16:13:07.044648 (+ 2012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:07.044654 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:07.044763 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:07.044877 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:07.047657 (+ 2780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:07.048730 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:07.051079 (+ 2349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:07.051573 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:07.051711 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:07.057041 (+ 5330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:13:07.057050 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:13:07.057056 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:13:07.057108 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:13:07.061079 (+ 3971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:07.062353 (+ 1274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:07.062357 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:07.062359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:07.062503 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:07.063129 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:07.063133 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:07.063136 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:07.063547 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:13:07.064760 (+ 1213us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:07.064914 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 16:13:07.065137 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 16:13:07.066771 6290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:07.041591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:40365 (local address 127.29.234.3:36565)
0924 16:13:07.042451 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:07.042505 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:07.044374 (+ 1869us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:07.045211 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:07.045226 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:07.046321 (+ 1095us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:13:07.047424 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:07.047462 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:07.048904 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:07.048914 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:07.050894 (+ 1980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:07.050906 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:07.052029 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:07.052037 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:07.052231 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:07.052926 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:13:07.052975 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:13:07.056792 (+ 3817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:13:07.061300 (+ 4508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:13:07.061315 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:13:07.061335 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:13:07.062172 (+ 837us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:13:07.062651 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:13:07.062656 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:13:07.062660 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:13:07.062959 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:13:07.063709 (+ 750us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:13:07.063717 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:13:07.064185 (+ 468us) client_negotiation.cc:769] Sending connection context
0924 16:13:07.064520 (+ 335us) client_negotiation.cc:240] Negotiation successful
0924 16:13:07.065131 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":397,"threads_started":1}
I20260924 16:13:07.088925 6264 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.3:33413
I20260924 16:13:07.088951 6402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.3:33413 every 8 connection(s)
I20260924 16:13:07.089416 6005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:07.073447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.3:32853 (local address 127.29.234.62:40365)
0924 16:13:07.073632 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:07.073637 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:07.074432 (+ 795us) server_negotiation.cc:407] Connection header received
0924 16:13:07.074644 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:07.074650 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:07.074745 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:07.074844 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:07.076244 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:07.077219 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:07.079195 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:07.079934 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:07.080089 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:07.083212 (+ 3123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:13:07.083220 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:13:07.083227 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:13:07.083280 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:13:07.086663 (+ 3383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:07.087322 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:07.087327 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:07.087336 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:07.087456 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:07.087834 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:07.087839 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:07.087842 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:07.088228 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:13:07.088392 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:07.088814 (+ 422us) server_negotiation.cc:299] Negotiation successful
0924 16:13:07.089028 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 16:13:07.089546 6368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:07.073560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:40365 (local address 127.29.234.3:32853)
0924 16:13:07.074335 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:07.074351 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:07.074503 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:07.074913 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:07.074923 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:07.075595 (+ 672us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:13:07.076071 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:07.076088 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:07.077445 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:07.077451 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:07.079044 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:07.079056 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:07.080371 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:07.080380 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:07.080651 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:07.081381 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:13:07.081411 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:13:07.082993 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:13:07.086797 (+ 3804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:13:07.086802 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:13:07.086806 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:13:07.087196 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:13:07.087566 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:13:07.087574 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:13:07.087578 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:13:07.087705 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:13:07.088357 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:13:07.088362 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:13:07.088562 (+ 200us) client_negotiation.cc:769] Sending connection context
0924 16:13:07.088703 (+ 141us) client_negotiation.cc:240] Negotiation successful
0924 16:13:07.089055 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":485,"threads_started":1}
I20260924 16:13:07.090476 6264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
I20260924 16:13:07.093078 6403 heartbeater.cc:347] Connected to a master server at 127.29.234.62:40365
I20260924 16:13:07.093310 6403 heartbeater.cc:464] Registering TS with master...
I20260924 16:13:07.094080 6403 heartbeater.cc:511] Master 127.29.234.62:40365 requested a full tablet report, sending...
I20260924 16:13:07.095367 5908 ts_manager.cc:194] Registered new tserver with Master: a204f0e9eec340e1bd4d74175dd392a2 (127.29.234.3:33413)
I20260924 16:13:07.096163 5908 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.234.3@KRBTEST.COM'} at 127.29.234.3:36565
I20260924 16:13:07.098034 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 6264
I20260924 16:13:07.098196 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/minicluster-data/ts-2/wal/instance
I20260924 16:13:07.098762 30632 external_mini_cluster.cc:1428] Setting key cd9ae2a7c3ec5a011644739c9ffe7921
I20260924 16:13:07.101680 30632 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 16:13:07 dist-test-slave-v5j9 krb5kdc[5862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266387, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 16:13:07 dist-test-slave-v5j9 krb5kdc[5862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:13:07 dist-test-slave-v5j9 krb5kdc[5862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:13:07.584126 6260 heartbeater.cc:503] Master 127.29.234.62:40365 was elected leader, sending a full tablet report...
I20260924 16:13:08.098809 6403 heartbeater.cc:503] Master 127.29.234.62:40365 was elected leader, sending a full tablet report...
W20260924 16:13:08.251608 6411 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.105s user 0.005s sys 0.009s
W20260924 16:13:08.252897 6005 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:13:07.140713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49446 (local address 127.29.234.62:40365)
0924 16:13:07.140899 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:07.140904 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:07.141154 (+ 250us) server_negotiation.cc:407] Connection header received
0924 16:13:07.141307 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:07.141311 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:07.141446 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:07.141538 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:07.142851 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:07.143631 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:07.144795 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:07.145222 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:07.145314 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:08.252626 (+1107312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49446: BlockingRecv error: recv got EOF from 127.0.0.1:49446 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 24 16:13:08 dist-test-slave-v5j9 krb5kdc[5862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266387, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM
I20260924 16:13:08.278766 6005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:08.256498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49458 (local address 127.29.234.62:40365)
0924 16:13:08.256696 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:08.256701 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:08.256927 (+ 226us) server_negotiation.cc:407] Connection header received
0924 16:13:08.257061 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:08.257087 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:08.257182 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:08.257295 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:08.258523 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:08.259224 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:08.260439 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:08.260839 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:08.260931 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:08.273884 (+ 12953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:13:08.273890 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:13:08.273893 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:13:08.273931 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:13:08.276298 (+ 2367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:08.276895 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:08.276899 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:08.276901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:08.277033 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:08.277433 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:08.277436 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:08.277438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:08.277716 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:13:08.277915 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:08.278334 (+ 419us) server_negotiation.cc:299] Negotiation successful
0924 16:13:08.278507 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 16:13:08.283001 5908 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49458:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:13:08.289951 5908 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:13:08.315678 6425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:08.302734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:41013 (local address 127.0.0.1:49384)
0924 16:13:08.304710 (+ 1976us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:08.304732 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:08.304857 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:08.308257 (+ 3400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:08.308266 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:08.308298 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:13:08.308886 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:08.308897 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:08.310886 (+ 1989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:08.310891 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:08.313281 (+ 2390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:08.313293 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:08.314711 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:08.314718 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:08.314915 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:08.314971 (+ 56us) client_negotiation.cc:769] Sending connection context
0924 16:13:08.315088 (+ 117us) client_negotiation.cc:240] Negotiation successful
0924 16:13:08.315268 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1801,"mutex_wait_us":121,"thread_start_us":730,"threads_started":1}
I20260924 16:13:08.316015 6423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:08.302478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.3:33413 (local address 127.0.0.1:50420)
0924 16:13:08.303805 (+ 1327us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:08.303848 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:08.304003 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:08.308448 (+ 4445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:08.308458 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:08.308499 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:13:08.309160 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:08.309178 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:08.310727 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:08.310732 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:08.313175 (+ 2443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:08.313195 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:08.315037 (+ 1842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:08.315044 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:08.315222 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:08.315296 (+ 74us) client_negotiation.cc:769] Sending connection context
0924 16:13:08.315379 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 16:13:08.315503 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1078,"mutex_wait_us":54,"spinlock_wait_cycles":12160,"thread_start_us":5167,"threads_started":1}
I20260924 16:13:08.316107 6426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:08.304063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49384 (local address 127.29.234.1:41013)
0924 16:13:08.306769 (+ 2706us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:08.306780 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:08.306818 (+ 38us) server_negotiation.cc:407] Connection header received
0924 16:13:08.306950 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:08.306963 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:08.307298 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:08.308228 (+ 930us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:13:08.309117 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:08.310728 (+ 1611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:08.313417 (+ 2689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:08.314480 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:08.315387 (+ 907us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:08.315580 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:08.315661 (+ 81us) server_negotiation.cc:299] Negotiation successful
0924 16:13:08.315828 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2509,"thread_start_us":2332,"threads_started":1}
I20260924 16:13:08.316707 6424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:08.303143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50420 (local address 127.29.234.3:33413)
0924 16:13:08.307902 (+ 4759us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:08.307909 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:08.307931 (+ 22us) server_negotiation.cc:407] Connection header received
0924 16:13:08.308008 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:08.308013 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:08.308215 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:08.308624 (+ 409us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:13:08.309375 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:08.310577 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:08.313338 (+ 2761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:08.314810 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:08.315827 (+ 1017us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:08.316018 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:08.316093 (+ 75us) server_negotiation.cc:299] Negotiation successful
0924 16:13:08.316416 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4614,"thread_start_us":4501,"threads_started":1}
I20260924 16:13:08.316901 6427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:08.304533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56866 (local address 127.29.234.2:36571)
0924 16:13:08.306223 (+ 1690us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:08.306237 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:08.308054 (+ 1817us) server_negotiation.cc:407] Connection header received
0924 16:13:08.308323 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:08.308330 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:08.308523 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:08.308737 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:13:08.309592 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:08.311017 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:08.313125 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:08.314625 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:08.314762 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:08.314995 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:08.316511 (+ 1516us) server_negotiation.cc:299] Negotiation successful
0924 16:13:08.316642 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1459,"thread_start_us":1289,"threads_started":1}
I20260924 16:13:08.316919 6422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:08.303335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.2:36571 (local address 127.0.0.1:56866)
0924 16:13:08.307931 (+ 4596us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:08.307946 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:08.308092 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:08.308866 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:08.308870 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:08.308894 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:13:08.309399 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:08.309411 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:08.311219 (+ 1808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:08.311224 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:08.312964 (+ 1740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:08.312977 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:08.315124 (+ 2147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:08.315129 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:08.316310 (+ 1181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:08.316345 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 16:13:08.316436 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 16:13:08.316564 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4458,"mutex_wait_us":1015,"thread_start_us":1685,"threads_started":1}
I20260924 16:13:08.320813 6188 tablet_service.cc:1467] Processing CreateTablet for tablet 6a3d0f9452974de3a719c5e926fae4b2 (DEFAULT_TABLE table=test-table [id=5db3746cfae9447a9249f2b8297b7bec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:13:08.320813 6331 tablet_service.cc:1467] Processing CreateTablet for tablet 6a3d0f9452974de3a719c5e926fae4b2 (DEFAULT_TABLE table=test-table [id=5db3746cfae9447a9249f2b8297b7bec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:13:08.320791 6048 tablet_service.cc:1467] Processing CreateTablet for tablet 6a3d0f9452974de3a719c5e926fae4b2 (DEFAULT_TABLE table=test-table [id=5db3746cfae9447a9249f2b8297b7bec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:13:08.322090 6048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a3d0f9452974de3a719c5e926fae4b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:13:08.322162 6188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a3d0f9452974de3a719c5e926fae4b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:13:08.322165 6331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a3d0f9452974de3a719c5e926fae4b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:13:08.335069 6428 tablet_bootstrap.cc:492] T 6a3d0f9452974de3a719c5e926fae4b2 P 284ce6491d364d2a83dad33520490442: Bootstrap starting.
I20260924 16:13:08.336551 6429 tablet_bootstrap.cc:492] T 6a3d0f9452974de3a719c5e926fae4b2 P a204f0e9eec340e1bd4d74175dd392a2: Bootstrap starting.
I20260924 16:13:08.338862 6428 tablet_bootstrap.cc:654] T 6a3d0f9452974de3a719c5e926fae4b2 P 284ce6491d364d2a83dad33520490442: Neither blocks nor log segments found. Creating new log.
I20260924 16:13:08.339685 6430 tablet_bootstrap.cc:492] T 6a3d0f9452974de3a719c5e926fae4b2 P 43317d13dde547deabe618d1ea6110fa: Bootstrap starting.
I20260924 16:13:08.339950 6429 tablet_bootstrap.cc:654] T 6a3d0f9452974de3a719c5e926fae4b2 P a204f0e9eec340e1bd4d74175dd392a2: Neither blocks nor log segments found. Creating new log.
I20260924 16:13:08.340191 6428 log.cc:824] T 6a3d0f9452974de3a719c5e926fae4b2 P 284ce6491d364d2a83dad33520490442: Log is configured to *not* fsync() on all Append() calls
I20260924 16:13:08.341204 6429 log.cc:824] T 6a3d0f9452974de3a719c5e926fae4b2 P a204f0e9eec340e1bd4d74175dd392a2: Log is configured to *not* fsync() on all Append() calls
I20260924 16:13:08.343616 6428 tablet_bootstrap.cc:492] T 6a3d0f9452974de3a719c5e926fae4b2 P 284ce6491d364d2a83dad33520490442: No bootstrap required, opened a new log
I20260924 16:13:08.343644 6430 tablet_bootstrap.cc:654] T 6a3d0f9452974de3a719c5e926fae4b2 P 43317d13dde547deabe618d1ea6110fa: Neither blocks nor log segments found. Creating new log.
I20260924 16:13:08.343894 6428 ts_tablet_manager.cc:1397] T 6a3d0f9452974de3a719c5e926fae4b2 P 284ce6491d364d2a83dad33520490442: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260924 16:13:08.344134 6429 tablet_bootstrap.cc:492] T 6a3d0f9452974de3a719c5e926fae4b2 P a204f0e9eec340e1bd4d74175dd392a2: No bootstrap required, opened a new log
I20260924 16:13:08.344381 6429 ts_tablet_manager.cc:1397] T 6a3d0f9452974de3a719c5e926fae4b2 P a204f0e9eec340e1bd4d74175dd392a2: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260924 16:13:08.344802 6430 log.cc:824] T 6a3d0f9452974de3a719c5e926fae4b2 P 43317d13dde547deabe618d1ea6110fa: Log is configured to *not* fsync() on all Append() calls
I20260924 16:13:08.347615 6430 tablet_bootstrap.cc:492] T 6a3d0f9452974de3a719c5e926fae4b2 P 43317d13dde547deabe618d1ea6110fa: No bootstrap required, opened a new log
I20260924 16:13:08.347860 6430 ts_tablet_manager.cc:1397] T 6a3d0f9452974de3a719c5e926fae4b2 P 43317d13dde547deabe618d1ea6110fa: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260924 16:13:08.351220 6428 raft_consensus.cc:348] T 6a3d0f9452974de3a719c5e926fae4b2 P 284ce6491d364d2a83dad33520490442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a204f0e9eec340e1bd4d74175dd392a2" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 33413 } } peers { permanent_uuid: "284ce6491d364d2a83dad33520490442" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 41013 } } peers { permanent_uuid: "43317d13dde547deabe618d1ea6110fa" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36571 } }
I20260924 16:13:08.351601 6428 raft_consensus.cc:374] T 6a3d0f9452974de3a719c5e926fae4b2 P 284ce6491d364d2a83dad33520490442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:13:08.351725 6428 raft_consensus.cc:729] T 6a3d0f9452974de3a719c5e926fae4b2 P 284ce6491d364d2a83dad33520490442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 284ce6491d364d2a83dad33520490442, State: Initialized, Role: FOLLOWER
I20260924 16:13:08.352299 6428 consensus_queue.cc:260] T 6a3d0f9452974de3a719c5e926fae4b2 P 284ce6491d364d2a83dad33520490442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a204f0e9eec340e1bd4d74175dd392a2" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 33413 } } peers { permanent_uuid: "284ce6491d364d2a83dad33520490442" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 41013 } } peers { permanent_uuid: "43317d13dde547deabe618d1ea6110fa" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36571 } }
I20260924 16:13:08.354552 6429 raft_consensus.cc:348] T 6a3d0f9452974de3a719c5e926fae4b2 P a204f0e9eec340e1bd4d74175dd392a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a204f0e9eec340e1bd4d74175dd392a2" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 33413 } } peers { permanent_uuid: "284ce6491d364d2a83dad33520490442" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 41013 } } peers { permanent_uuid: "43317d13dde547deabe618d1ea6110fa" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36571 } }
I20260924 16:13:08.355031 6429 raft_consensus.cc:374] T 6a3d0f9452974de3a719c5e926fae4b2 P a204f0e9eec340e1bd4d74175dd392a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:13:08.355201 6429 raft_consensus.cc:729] T 6a3d0f9452974de3a719c5e926fae4b2 P a204f0e9eec340e1bd4d74175dd392a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a204f0e9eec340e1bd4d74175dd392a2, State: Initialized, Role: FOLLOWER
I20260924 16:13:08.355511 6428 ts_tablet_manager.cc:1428] T 6a3d0f9452974de3a719c5e926fae4b2 P 284ce6491d364d2a83dad33520490442: Time spent starting tablet: real 0.011s user 0.010s sys 0.003s
I20260924 16:13:08.355756 6429 consensus_queue.cc:260] T 6a3d0f9452974de3a719c5e926fae4b2 P a204f0e9eec340e1bd4d74175dd392a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a204f0e9eec340e1bd4d74175dd392a2" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 33413 } } peers { permanent_uuid: "284ce6491d364d2a83dad33520490442" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 41013 } } peers { permanent_uuid: "43317d13dde547deabe618d1ea6110fa" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36571 } }
I20260924 16:13:08.357496 6429 ts_tablet_manager.cc:1428] T 6a3d0f9452974de3a719c5e926fae4b2 P a204f0e9eec340e1bd4d74175dd392a2: Time spent starting tablet: real 0.013s user 0.015s sys 0.000s
I20260924 16:13:08.358536 6430 raft_consensus.cc:348] T 6a3d0f9452974de3a719c5e926fae4b2 P 43317d13dde547deabe618d1ea6110fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a204f0e9eec340e1bd4d74175dd392a2" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 33413 } } peers { permanent_uuid: "284ce6491d364d2a83dad33520490442" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 41013 } } peers { permanent_uuid: "43317d13dde547deabe618d1ea6110fa" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36571 } }
I20260924 16:13:08.358948 6430 raft_consensus.cc:374] T 6a3d0f9452974de3a719c5e926fae4b2 P 43317d13dde547deabe618d1ea6110fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:13:08.359054 6430 raft_consensus.cc:729] T 6a3d0f9452974de3a719c5e926fae4b2 P 43317d13dde547deabe618d1ea6110fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43317d13dde547deabe618d1ea6110fa, State: Initialized, Role: FOLLOWER
I20260924 16:13:08.359712 6430 consensus_queue.cc:260] T 6a3d0f9452974de3a719c5e926fae4b2 P 43317d13dde547deabe618d1ea6110fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a204f0e9eec340e1bd4d74175dd392a2" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 33413 } } peers { permanent_uuid: "284ce6491d364d2a83dad33520490442" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 41013 } } peers { permanent_uuid: "43317d13dde547deabe618d1ea6110fa" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36571 } }
I20260924 16:13:08.361550 6430 ts_tablet_manager.cc:1428] T 6a3d0f9452974de3a719c5e926fae4b2 P 43317d13dde547deabe618d1ea6110fa: Time spent starting tablet: real 0.014s user 0.013s sys 0.002s
I20260924 16:13:08.397342 6436 raft_consensus.cc:482] T 6a3d0f9452974de3a719c5e926fae4b2 P a204f0e9eec340e1bd4d74175dd392a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:13:08.397740 6436 raft_consensus.cc:504] T 6a3d0f9452974de3a719c5e926fae4b2 P a204f0e9eec340e1bd4d74175dd392a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a204f0e9eec340e1bd4d74175dd392a2" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 33413 } } peers { permanent_uuid: "284ce6491d364d2a83dad33520490442" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 41013 } } peers { permanent_uuid: "43317d13dde547deabe618d1ea6110fa" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36571 } }
I20260924 16:13:08.399518 6436 leader_election.cc:288] T 6a3d0f9452974de3a719c5e926fae4b2 P a204f0e9eec340e1bd4d74175dd392a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 284ce6491d364d2a83dad33520490442 (127.29.234.1:41013), 43317d13dde547deabe618d1ea6110fa (127.29.234.2:36571)
I20260924 16:13:08.406744 6427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:08.400001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.3:56249 (local address 127.29.234.2:36571)
0924 16:13:08.400219 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:08.400227 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:08.401205 (+ 978us) server_negotiation.cc:407] Connection header received
0924 16:13:08.401468 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:08.401473 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:08.401597 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:08.401684 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:13:08.402471 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:08.403188 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:08.404950 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:08.405821 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:08.406004 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:08.406073 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:08.406382 (+ 309us) server_negotiation.cc:299] Negotiation successful
0924 16:13:08.406496 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 16:13:08.406754 6426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:08.400094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.3:56141 (local address 127.29.234.1:41013)
0924 16:13:08.400339 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:08.400348 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:08.401110 (+ 762us) server_negotiation.cc:407] Connection header received
0924 16:13:08.401379 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:08.401387 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:08.401481 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:08.401607 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:13:08.402370 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:08.403105 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:08.404994 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:08.405786 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:08.405939 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:08.406028 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:08.406344 (+ 316us) server_negotiation.cc:299] Negotiation successful
0924 16:13:08.406472 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260924 16:13:08.406965 6438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:08.399940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:41013 (local address 127.29.234.3:56141)
0924 16:13:08.400987 (+ 1047us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:08.401007 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:08.401208 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:08.401647 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:08.401651 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:08.401678 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:13:08.402107 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:08.402123 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:08.403241 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:08.403248 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:08.404823 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:08.404835 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:08.406010 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:08.406017 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:08.406189 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:08.406213 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 16:13:08.406289 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 16:13:08.406433 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":882,"mutex_wait_us":4,"thread_start_us":640,"threads_started":1}
I20260924 16:13:08.406994 6439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:08.400020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.2:36571 (local address 127.29.234.3:56249)
0924 16:13:08.401099 (+ 1079us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:08.401117 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:08.401255 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:08.401806 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:08.401811 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:08.401848 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:13:08.402292 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:08.402308 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:08.403325 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:08.403329 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:08.404799 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:08.404813 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:08.406067 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:08.406074 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:08.406211 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:08.406233 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 16:13:08.406341 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 16:13:08.406472 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":917,"mutex_wait_us":2,"thread_start_us":721,"threads_started":1}
I20260924 16:13:08.408051 6208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a3d0f9452974de3a719c5e926fae4b2" candidate_uuid: "a204f0e9eec340e1bd4d74175dd392a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43317d13dde547deabe618d1ea6110fa" is_pre_election: true
I20260924 16:13:08.408279 6073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a3d0f9452974de3a719c5e926fae4b2" candidate_uuid: "a204f0e9eec340e1bd4d74175dd392a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "284ce6491d364d2a83dad33520490442" is_pre_election: true
I20260924 16:13:08.408578 6208 raft_consensus.cc:2454] T 6a3d0f9452974de3a719c5e926fae4b2 P 43317d13dde547deabe618d1ea6110fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a204f0e9eec340e1bd4d74175dd392a2 in term 0.
I20260924 16:13:08.408838 6073 raft_consensus.cc:2454] T 6a3d0f9452974de3a719c5e926fae4b2 P 284ce6491d364d2a83dad33520490442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a204f0e9eec340e1bd4d74175dd392a2 in term 0.
I20260924 16:13:08.409271 6284 leader_election.cc:302] T 6a3d0f9452974de3a719c5e926fae4b2 P a204f0e9eec340e1bd4d74175dd392a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43317d13dde547deabe618d1ea6110fa, a204f0e9eec340e1bd4d74175dd392a2; no voters:
I20260924 16:13:08.409713 6436 raft_consensus.cc:2781] T 6a3d0f9452974de3a719c5e926fae4b2 P a204f0e9eec340e1bd4d74175dd392a2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:13:08.409876 6436 raft_consensus.cc:482] T 6a3d0f9452974de3a719c5e926fae4b2 P a204f0e9eec340e1bd4d74175dd392a2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:13:08.409984 6436 raft_consensus.cc:3037] T 6a3d0f9452974de3a719c5e926fae4b2 P a204f0e9eec340e1bd4d74175dd392a2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:13:08.412081 6436 raft_consensus.cc:504] T 6a3d0f9452974de3a719c5e926fae4b2 P a204f0e9eec340e1bd4d74175dd392a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a204f0e9eec340e1bd4d74175dd392a2" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 33413 } } peers { permanent_uuid: "284ce6491d364d2a83dad33520490442" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 41013 } } peers { permanent_uuid: "43317d13dde547deabe618d1ea6110fa" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36571 } }
I20260924 16:13:08.412623 6436 leader_election.cc:288] T 6a3d0f9452974de3a719c5e926fae4b2 P a204f0e9eec340e1bd4d74175dd392a2 [CANDIDATE]: Term 1 election: Requested vote from peers 284ce6491d364d2a83dad33520490442 (127.29.234.1:41013), 43317d13dde547deabe618d1ea6110fa (127.29.234.2:36571)
I20260924 16:13:08.413025 6073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a3d0f9452974de3a719c5e926fae4b2" candidate_uuid: "a204f0e9eec340e1bd4d74175dd392a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "284ce6491d364d2a83dad33520490442"
I20260924 16:13:08.413087 6208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a3d0f9452974de3a719c5e926fae4b2" candidate_uuid: "a204f0e9eec340e1bd4d74175dd392a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43317d13dde547deabe618d1ea6110fa"
I20260924 16:13:08.413234 6073 raft_consensus.cc:3037] T 6a3d0f9452974de3a719c5e926fae4b2 P 284ce6491d364d2a83dad33520490442 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:13:08.413275 6208 raft_consensus.cc:3037] T 6a3d0f9452974de3a719c5e926fae4b2 P 43317d13dde547deabe618d1ea6110fa [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:13:08.415555 6208 raft_consensus.cc:2454] T 6a3d0f9452974de3a719c5e926fae4b2 P 43317d13dde547deabe618d1ea6110fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a204f0e9eec340e1bd4d74175dd392a2 in term 1.
I20260924 16:13:08.415580 6073 raft_consensus.cc:2454] T 6a3d0f9452974de3a719c5e926fae4b2 P 284ce6491d364d2a83dad33520490442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a204f0e9eec340e1bd4d74175dd392a2 in term 1.
I20260924 16:13:08.415954 6284 leader_election.cc:302] T 6a3d0f9452974de3a719c5e926fae4b2 P a204f0e9eec340e1bd4d74175dd392a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43317d13dde547deabe618d1ea6110fa, a204f0e9eec340e1bd4d74175dd392a2; no voters:
I20260924 16:13:08.416216 6436 raft_consensus.cc:2781] T 6a3d0f9452974de3a719c5e926fae4b2 P a204f0e9eec340e1bd4d74175dd392a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:13:08.416661 6436 raft_consensus.cc:686] T 6a3d0f9452974de3a719c5e926fae4b2 P a204f0e9eec340e1bd4d74175dd392a2 [term 1 LEADER]: Becoming Leader. State: Replica: a204f0e9eec340e1bd4d74175dd392a2, State: Running, Role: LEADER
I20260924 16:13:08.417129 6436 consensus_queue.cc:237] T 6a3d0f9452974de3a719c5e926fae4b2 P a204f0e9eec340e1bd4d74175dd392a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a204f0e9eec340e1bd4d74175dd392a2" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 33413 } } peers { permanent_uuid: "284ce6491d364d2a83dad33520490442" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 41013 } } peers { permanent_uuid: "43317d13dde547deabe618d1ea6110fa" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36571 } }
I20260924 16:13:08.421713 5907 catalog_manager.cc:5495] T 6a3d0f9452974de3a719c5e926fae4b2 P a204f0e9eec340e1bd4d74175dd392a2 reported cstate change: term changed from 0 to 1, leader changed from <none> to a204f0e9eec340e1bd4d74175dd392a2 (127.29.234.3). New cstate: current_term: 1 leader_uuid: "a204f0e9eec340e1bd4d74175dd392a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a204f0e9eec340e1bd4d74175dd392a2" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 33413 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "284ce6491d364d2a83dad33520490442" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 41013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43317d13dde547deabe618d1ea6110fa" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36571 } health_report { overall_health: UNKNOWN } } }
I20260924 16:13:08.437980 6424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:08.432919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50428 (local address 127.29.234.3:33413)
0924 16:13:08.433147 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:08.433153 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:08.433182 (+ 29us) server_negotiation.cc:407] Connection header received
0924 16:13:08.433443 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:08.433447 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:08.433545 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:08.433671 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:13:08.433976 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:08.434661 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:08.435906 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:08.436294 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:08.436390 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:08.436741 (+ 351us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:13:08.437313 (+ 572us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:13:08.437481 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:08.437580 (+ 99us) server_negotiation.cc:299] Negotiation successful
0924 16:13:08.437675 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260924 16:13:08.445636 6208 raft_consensus.cc:1261] T 6a3d0f9452974de3a719c5e926fae4b2 P 43317d13dde547deabe618d1ea6110fa [term 1 FOLLOWER]: Refusing update from remote peer a204f0e9eec340e1bd4d74175dd392a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:13:08.445742 6073 raft_consensus.cc:1261] T 6a3d0f9452974de3a719c5e926fae4b2 P 284ce6491d364d2a83dad33520490442 [term 1 FOLLOWER]: Refusing update from remote peer a204f0e9eec340e1bd4d74175dd392a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:13:08.446664 6440 consensus_queue.cc:1035] T 6a3d0f9452974de3a719c5e926fae4b2 P a204f0e9eec340e1bd4d74175dd392a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43317d13dde547deabe618d1ea6110fa" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:13:08.447006 6436 consensus_queue.cc:1035] T 6a3d0f9452974de3a719c5e926fae4b2 P a204f0e9eec340e1bd4d74175dd392a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "284ce6491d364d2a83dad33520490442" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 41013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:13:08.476814 5907 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49458:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:13:08.477087 5907 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49458:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:13:08.480885 5907 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 441441046a974081ab6e0e4a1576c8f7: Sending DeleteTablet for 3 replicas of tablet 6a3d0f9452974de3a719c5e926fae4b2
I20260924 16:13:08.482074 6331 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a3d0f9452974de3a719c5e926fae4b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:13:08 UTC) from {username='oryx', principal='oryx/127.29.234.62@KRBTEST.COM'} at 127.0.0.1:50420
I20260924 16:13:08.482165 6048 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a3d0f9452974de3a719c5e926fae4b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:13:08 UTC) from {username='oryx', principal='oryx/127.29.234.62@KRBTEST.COM'} at 127.0.0.1:49384
I20260924 16:13:08.482220 6188 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a3d0f9452974de3a719c5e926fae4b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:13:08 UTC) from {username='oryx', principal='oryx/127.29.234.62@KRBTEST.COM'} at 127.0.0.1:56866
I20260924 16:13:08.483771 5907 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49458:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:13:08.484661 5907 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:13:08.488458 6456 tablet_replica.cc:330] T 6a3d0f9452974de3a719c5e926fae4b2 P 43317d13dde547deabe618d1ea6110fa: stopping tablet replica
I20260924 16:13:08.489117 6456 raft_consensus.cc:2229] T 6a3d0f9452974de3a719c5e926fae4b2 P 43317d13dde547deabe618d1ea6110fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:13:08.489807 6456 raft_consensus.cc:2258] T 6a3d0f9452974de3a719c5e926fae4b2 P 43317d13dde547deabe618d1ea6110fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:13:08.496670 6047 tablet_service.cc:1467] Processing CreateTablet for tablet 7333ffb3d6eb4e5bab2d0e3bb5c92764 (DEFAULT_TABLE table=test-table [id=f9fc73d97bcf4087ae0d8787d5ec6c7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:13:08.497471 6047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7333ffb3d6eb4e5bab2d0e3bb5c92764. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:13:08.496285 6456 ts_tablet_manager.cc:1905] T 6a3d0f9452974de3a719c5e926fae4b2 P 43317d13dde547deabe618d1ea6110fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:13:08.502920 6330 tablet_service.cc:1467] Processing CreateTablet for tablet 7333ffb3d6eb4e5bab2d0e3bb5c92764 (DEFAULT_TABLE table=test-table [id=f9fc73d97bcf4087ae0d8787d5ec6c7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:13:08.503742 6330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7333ffb3d6eb4e5bab2d0e3bb5c92764. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:13:08.508701 6456 ts_tablet_manager.cc:1918] T 6a3d0f9452974de3a719c5e926fae4b2 P 43317d13dde547deabe618d1ea6110fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:13:08.513914 6456 log.cc:1196] T 6a3d0f9452974de3a719c5e926fae4b2 P 43317d13dde547deabe618d1ea6110fa: Deleting WAL directory at /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/minicluster-data/ts-1/wal/wals/6a3d0f9452974de3a719c5e926fae4b2
I20260924 16:13:08.513871 6188 tablet_service.cc:1467] Processing CreateTablet for tablet 7333ffb3d6eb4e5bab2d0e3bb5c92764 (DEFAULT_TABLE table=test-table [id=f9fc73d97bcf4087ae0d8787d5ec6c7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:13:08.514632 6456 ts_tablet_manager.cc:1939] T 6a3d0f9452974de3a719c5e926fae4b2 P 43317d13dde547deabe618d1ea6110fa: Deleting consensus metadata
I20260924 16:13:08.514696 6188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7333ffb3d6eb4e5bab2d0e3bb5c92764. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:13:08.516902 5893 catalog_manager.cc:4841] TS 43317d13dde547deabe618d1ea6110fa (127.29.234.2:36571): tablet 6a3d0f9452974de3a719c5e926fae4b2 (table test-table [id=5db3746cfae9447a9249f2b8297b7bec]) successfully deleted
I20260924 16:13:08.520584 6428 tablet_bootstrap.cc:492] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P 284ce6491d364d2a83dad33520490442: Bootstrap starting.
I20260924 16:13:08.522488 6428 tablet_bootstrap.cc:654] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P 284ce6491d364d2a83dad33520490442: Neither blocks nor log segments found. Creating new log.
I20260924 16:13:08.523083 6429 tablet_bootstrap.cc:492] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P a204f0e9eec340e1bd4d74175dd392a2: Bootstrap starting.
I20260924 16:13:08.525363 6429 tablet_bootstrap.cc:654] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P a204f0e9eec340e1bd4d74175dd392a2: Neither blocks nor log segments found. Creating new log.
I20260924 16:13:08.528795 6454 tablet_replica.cc:330] T 6a3d0f9452974de3a719c5e926fae4b2 P a204f0e9eec340e1bd4d74175dd392a2: stopping tablet replica
I20260924 16:13:08.530231 6428 tablet_bootstrap.cc:492] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P 284ce6491d364d2a83dad33520490442: No bootstrap required, opened a new log
I20260924 16:13:08.530407 6428 ts_tablet_manager.cc:1397] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P 284ce6491d364d2a83dad33520490442: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.006s
I20260924 16:13:08.531430 6428 raft_consensus.cc:348] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P 284ce6491d364d2a83dad33520490442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284ce6491d364d2a83dad33520490442" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 41013 } } peers { permanent_uuid: "43317d13dde547deabe618d1ea6110fa" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36571 } } peers { permanent_uuid: "a204f0e9eec340e1bd4d74175dd392a2" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 33413 } }
I20260924 16:13:08.531697 6428 raft_consensus.cc:374] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P 284ce6491d364d2a83dad33520490442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:13:08.531792 6428 raft_consensus.cc:729] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P 284ce6491d364d2a83dad33520490442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 284ce6491d364d2a83dad33520490442, State: Initialized, Role: FOLLOWER
I20260924 16:13:08.532189 6428 consensus_queue.cc:260] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P 284ce6491d364d2a83dad33520490442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284ce6491d364d2a83dad33520490442" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 41013 } } peers { permanent_uuid: "43317d13dde547deabe618d1ea6110fa" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36571 } } peers { permanent_uuid: "a204f0e9eec340e1bd4d74175dd392a2" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 33413 } }
I20260924 16:13:08.532948 6428 ts_tablet_manager.cc:1428] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P 284ce6491d364d2a83dad33520490442: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 16:13:08.533205 6430 tablet_bootstrap.cc:492] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P 43317d13dde547deabe618d1ea6110fa: Bootstrap starting.
I20260924 16:13:08.533716 6454 raft_consensus.cc:2229] T 6a3d0f9452974de3a719c5e926fae4b2 P a204f0e9eec340e1bd4d74175dd392a2 [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:13:08.534606 6454 raft_consensus.cc:2258] T 6a3d0f9452974de3a719c5e926fae4b2 P a204f0e9eec340e1bd4d74175dd392a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:13:08.534878 6430 tablet_bootstrap.cc:654] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P 43317d13dde547deabe618d1ea6110fa: Neither blocks nor log segments found. Creating new log.
I20260924 16:13:08.535450 6455 tablet_replica.cc:330] T 6a3d0f9452974de3a719c5e926fae4b2 P 284ce6491d364d2a83dad33520490442: stopping tablet replica
I20260924 16:13:08.535912 6455 raft_consensus.cc:2229] T 6a3d0f9452974de3a719c5e926fae4b2 P 284ce6491d364d2a83dad33520490442 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:13:08.536413 6455 raft_consensus.cc:2258] T 6a3d0f9452974de3a719c5e926fae4b2 P 284ce6491d364d2a83dad33520490442 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:13:08.538587 6455 ts_tablet_manager.cc:1905] T 6a3d0f9452974de3a719c5e926fae4b2 P 284ce6491d364d2a83dad33520490442: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:13:08.538674 6429 tablet_bootstrap.cc:492] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P a204f0e9eec340e1bd4d74175dd392a2: No bootstrap required, opened a new log
I20260924 16:13:08.538822 6429 ts_tablet_manager.cc:1397] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P a204f0e9eec340e1bd4d74175dd392a2: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.009s
I20260924 16:13:08.538847 6454 ts_tablet_manager.cc:1905] T 6a3d0f9452974de3a719c5e926fae4b2 P a204f0e9eec340e1bd4d74175dd392a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:13:08.539733 6429 raft_consensus.cc:348] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P a204f0e9eec340e1bd4d74175dd392a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284ce6491d364d2a83dad33520490442" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 41013 } } peers { permanent_uuid: "43317d13dde547deabe618d1ea6110fa" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36571 } } peers { permanent_uuid: "a204f0e9eec340e1bd4d74175dd392a2" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 33413 } }
I20260924 16:13:08.539968 6429 raft_consensus.cc:374] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P a204f0e9eec340e1bd4d74175dd392a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:13:08.540077 6429 raft_consensus.cc:729] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P a204f0e9eec340e1bd4d74175dd392a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a204f0e9eec340e1bd4d74175dd392a2, State: Initialized, Role: FOLLOWER
I20260924 16:13:08.540386 6429 consensus_queue.cc:260] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P a204f0e9eec340e1bd4d74175dd392a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284ce6491d364d2a83dad33520490442" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 41013 } } peers { permanent_uuid: "43317d13dde547deabe618d1ea6110fa" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36571 } } peers { permanent_uuid: "a204f0e9eec340e1bd4d74175dd392a2" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 33413 } }
I20260924 16:13:08.541886 6429 ts_tablet_manager.cc:1428] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P a204f0e9eec340e1bd4d74175dd392a2: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260924 16:13:08.543483 6430 tablet_bootstrap.cc:492] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P 43317d13dde547deabe618d1ea6110fa: No bootstrap required, opened a new log
I20260924 16:13:08.543635 6430 ts_tablet_manager.cc:1397] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P 43317d13dde547deabe618d1ea6110fa: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.004s
I20260924 16:13:08.544556 6430 raft_consensus.cc:348] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P 43317d13dde547deabe618d1ea6110fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284ce6491d364d2a83dad33520490442" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 41013 } } peers { permanent_uuid: "43317d13dde547deabe618d1ea6110fa" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36571 } } peers { permanent_uuid: "a204f0e9eec340e1bd4d74175dd392a2" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 33413 } }
I20260924 16:13:08.544842 6430 raft_consensus.cc:374] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P 43317d13dde547deabe618d1ea6110fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:13:08.544981 6430 raft_consensus.cc:729] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P 43317d13dde547deabe618d1ea6110fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43317d13dde547deabe618d1ea6110fa, State: Initialized, Role: FOLLOWER
I20260924 16:13:08.545404 6430 consensus_queue.cc:260] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P 43317d13dde547deabe618d1ea6110fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284ce6491d364d2a83dad33520490442" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 41013 } } peers { permanent_uuid: "43317d13dde547deabe618d1ea6110fa" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36571 } } peers { permanent_uuid: "a204f0e9eec340e1bd4d74175dd392a2" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 33413 } }
I20260924 16:13:08.546082 6430 ts_tablet_manager.cc:1428] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P 43317d13dde547deabe618d1ea6110fa: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 16:13:08.547916 6455 ts_tablet_manager.cc:1918] T 6a3d0f9452974de3a719c5e926fae4b2 P 284ce6491d364d2a83dad33520490442: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:13:08.548120 6455 log.cc:1196] T 6a3d0f9452974de3a719c5e926fae4b2 P 284ce6491d364d2a83dad33520490442: Deleting WAL directory at /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/minicluster-data/ts-0/wal/wals/6a3d0f9452974de3a719c5e926fae4b2
I20260924 16:13:08.548720 6455 ts_tablet_manager.cc:1939] T 6a3d0f9452974de3a719c5e926fae4b2 P 284ce6491d364d2a83dad33520490442: Deleting consensus metadata
I20260924 16:13:08.549003 6454 ts_tablet_manager.cc:1918] T 6a3d0f9452974de3a719c5e926fae4b2 P a204f0e9eec340e1bd4d74175dd392a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:13:08.549235 6454 log.cc:1196] T 6a3d0f9452974de3a719c5e926fae4b2 P a204f0e9eec340e1bd4d74175dd392a2: Deleting WAL directory at /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/minicluster-data/ts-2/wal/wals/6a3d0f9452974de3a719c5e926fae4b2
I20260924 16:13:08.549866 6454 ts_tablet_manager.cc:1939] T 6a3d0f9452974de3a719c5e926fae4b2 P a204f0e9eec340e1bd4d74175dd392a2: Deleting consensus metadata
I20260924 16:13:08.550626 5892 catalog_manager.cc:4841] TS 284ce6491d364d2a83dad33520490442 (127.29.234.1:41013): tablet 6a3d0f9452974de3a719c5e926fae4b2 (table test-table [id=5db3746cfae9447a9249f2b8297b7bec]) successfully deleted
I20260924 16:13:08.551494 5891 catalog_manager.cc:4841] TS a204f0e9eec340e1bd4d74175dd392a2 (127.29.234.3:33413): tablet 6a3d0f9452974de3a719c5e926fae4b2 (table test-table [id=5db3746cfae9447a9249f2b8297b7bec]) successfully deleted
W20260924 16:13:08.557189 6120 tablet.cc:2369] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P 284ce6491d364d2a83dad33520490442: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 16:13:08.581679 6261 tablet.cc:2369] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P 43317d13dde547deabe618d1ea6110fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 16:13:08.593137 6404 tablet.cc:2369] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P a204f0e9eec340e1bd4d74175dd392a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:13:08.868543 6437 raft_consensus.cc:482] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P 43317d13dde547deabe618d1ea6110fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:13:08.868811 6437 raft_consensus.cc:504] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P 43317d13dde547deabe618d1ea6110fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284ce6491d364d2a83dad33520490442" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 41013 } } peers { permanent_uuid: "43317d13dde547deabe618d1ea6110fa" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36571 } } peers { permanent_uuid: "a204f0e9eec340e1bd4d74175dd392a2" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 33413 } }
I20260924 16:13:08.870622 6437 leader_election.cc:288] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P 43317d13dde547deabe618d1ea6110fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 284ce6491d364d2a83dad33520490442 (127.29.234.1:41013), a204f0e9eec340e1bd4d74175dd392a2 (127.29.234.3:33413)
I20260924 16:13:08.879366 6440 raft_consensus.cc:482] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P a204f0e9eec340e1bd4d74175dd392a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:13:08.879559 6426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:08.871318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.2:60915 (local address 127.29.234.1:41013)
0924 16:13:08.871562 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:08.871570 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:08.872540 (+ 970us) server_negotiation.cc:407] Connection header received
0924 16:13:08.872865 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:08.872870 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:08.872958 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:08.873028 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:13:08.873938 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:08.874634 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:08.877224 (+ 2590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:08.878521 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:08.878706 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:08.878782 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:08.879062 (+ 280us) server_negotiation.cc:299] Negotiation successful
0924 16:13:08.879266 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260924 16:13:08.879766 6461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:08.870893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:41013 (local address 127.29.234.2:60915)
0924 16:13:08.872364 (+ 1471us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:08.872386 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:08.872693 (+ 307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:08.873287 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:08.873292 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:08.873316 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:13:08.873807 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:08.873820 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:08.874766 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:08.874774 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:08.877052 (+ 2278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:08.877067 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:08.878745 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:08.878751 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:08.878903 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:08.878932 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 16:13:08.879022 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 16:13:08.879356 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1320,"mutex_wait_us":43,"thread_start_us":1458,"threads_started":1}
I20260924 16:13:08.879575 6440 raft_consensus.cc:504] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P a204f0e9eec340e1bd4d74175dd392a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284ce6491d364d2a83dad33520490442" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 41013 } } peers { permanent_uuid: "43317d13dde547deabe618d1ea6110fa" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36571 } } peers { permanent_uuid: "a204f0e9eec340e1bd4d74175dd392a2" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 33413 } }
I20260924 16:13:08.880332 6424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:08.871758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.2:54851 (local address 127.29.234.3:33413)
0924 16:13:08.871950 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:08.871956 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:08.872781 (+ 825us) server_negotiation.cc:407] Connection header received
0924 16:13:08.872918 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:08.872924 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:08.872998 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:08.873087 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:13:08.873803 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:08.874930 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:08.877267 (+ 2337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:08.878135 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:08.878354 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:08.878440 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:08.878768 (+ 328us) server_negotiation.cc:299] Negotiation successful
0924 16:13:08.878908 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 16:13:08.880477 6073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7333ffb3d6eb4e5bab2d0e3bb5c92764" candidate_uuid: "43317d13dde547deabe618d1ea6110fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "284ce6491d364d2a83dad33520490442" is_pre_election: true
I20260924 16:13:08.880736 6073 raft_consensus.cc:2454] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P 284ce6491d364d2a83dad33520490442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43317d13dde547deabe618d1ea6110fa in term 0.
I20260924 16:13:08.880847 6460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:08.871001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.3:33413 (local address 127.29.234.2:54851)
0924 16:13:08.872683 (+ 1682us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:08.872697 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:08.872799 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:08.873138 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:08.873142 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:08.873168 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:13:08.873598 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:08.873614 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:08.875063 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:08.875068 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:08.877053 (+ 1985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:08.877067 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:08.878372 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:08.878378 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:08.878550 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:08.878573 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 16:13:08.880297 (+ 1724us) client_negotiation.cc:240] Negotiation successful
0924 16:13:08.880426 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1572,"mutex_wait_us":4,"thread_start_us":999,"threads_started":1}
I20260924 16:13:08.880990 6072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7333ffb3d6eb4e5bab2d0e3bb5c92764" candidate_uuid: "a204f0e9eec340e1bd4d74175dd392a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "284ce6491d364d2a83dad33520490442" is_pre_election: true
I20260924 16:13:08.881135 6208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7333ffb3d6eb4e5bab2d0e3bb5c92764" candidate_uuid: "a204f0e9eec340e1bd4d74175dd392a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43317d13dde547deabe618d1ea6110fa" is_pre_election: true
I20260924 16:13:08.881278 6072 raft_consensus.cc:2454] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P 284ce6491d364d2a83dad33520490442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a204f0e9eec340e1bd4d74175dd392a2 in term 0.
I20260924 16:13:08.881408 6208 raft_consensus.cc:2454] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P 43317d13dde547deabe618d1ea6110fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a204f0e9eec340e1bd4d74175dd392a2 in term 0.
I20260924 16:13:08.881635 6440 leader_election.cc:288] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P a204f0e9eec340e1bd4d74175dd392a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 284ce6491d364d2a83dad33520490442 (127.29.234.1:41013), 43317d13dde547deabe618d1ea6110fa (127.29.234.2:36571)
I20260924 16:13:08.881820 6283 leader_election.cc:302] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P a204f0e9eec340e1bd4d74175dd392a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 284ce6491d364d2a83dad33520490442, a204f0e9eec340e1bd4d74175dd392a2; no voters:
I20260924 16:13:08.882185 6440 raft_consensus.cc:2781] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P a204f0e9eec340e1bd4d74175dd392a2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:13:08.882302 6440 raft_consensus.cc:482] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P a204f0e9eec340e1bd4d74175dd392a2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:13:08.882107 6351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7333ffb3d6eb4e5bab2d0e3bb5c92764" candidate_uuid: "43317d13dde547deabe618d1ea6110fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a204f0e9eec340e1bd4d74175dd392a2" is_pre_election: true
I20260924 16:13:08.882232 6140 leader_election.cc:302] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P 43317d13dde547deabe618d1ea6110fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 284ce6491d364d2a83dad33520490442, 43317d13dde547deabe618d1ea6110fa; no voters:
I20260924 16:13:08.882413 6440 raft_consensus.cc:3037] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P a204f0e9eec340e1bd4d74175dd392a2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:13:08.882640 6437 raft_consensus.cc:2781] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P 43317d13dde547deabe618d1ea6110fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:13:08.882771 6437 raft_consensus.cc:482] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P 43317d13dde547deabe618d1ea6110fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:13:08.882869 6437 raft_consensus.cc:3037] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P 43317d13dde547deabe618d1ea6110fa [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:13:08.885754 6440 raft_consensus.cc:504] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P a204f0e9eec340e1bd4d74175dd392a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284ce6491d364d2a83dad33520490442" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 41013 } } peers { permanent_uuid: "43317d13dde547deabe618d1ea6110fa" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36571 } } peers { permanent_uuid: "a204f0e9eec340e1bd4d74175dd392a2" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 33413 } }
I20260924 16:13:08.885890 6437 raft_consensus.cc:504] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P 43317d13dde547deabe618d1ea6110fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284ce6491d364d2a83dad33520490442" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 41013 } } peers { permanent_uuid: "43317d13dde547deabe618d1ea6110fa" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36571 } } peers { permanent_uuid: "a204f0e9eec340e1bd4d74175dd392a2" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 33413 } }
I20260924 16:13:08.886272 6351 raft_consensus.cc:2379] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P a204f0e9eec340e1bd4d74175dd392a2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 43317d13dde547deabe618d1ea6110fa in current term 1: Already voted for candidate a204f0e9eec340e1bd4d74175dd392a2 in this term.
I20260924 16:13:08.886823 6072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7333ffb3d6eb4e5bab2d0e3bb5c92764" candidate_uuid: "a204f0e9eec340e1bd4d74175dd392a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "284ce6491d364d2a83dad33520490442"
I20260924 16:13:08.887003 6072 raft_consensus.cc:3037] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P 284ce6491d364d2a83dad33520490442 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:13:08.887593 6351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7333ffb3d6eb4e5bab2d0e3bb5c92764" candidate_uuid: "43317d13dde547deabe618d1ea6110fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a204f0e9eec340e1bd4d74175dd392a2"
I20260924 16:13:08.887758 6437 leader_election.cc:288] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P 43317d13dde547deabe618d1ea6110fa [CANDIDATE]: Term 1 election: Requested vote from peers 284ce6491d364d2a83dad33520490442 (127.29.234.1:41013), a204f0e9eec340e1bd4d74175dd392a2 (127.29.234.3:33413)
I20260924 16:13:08.887828 6351 raft_consensus.cc:2379] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P a204f0e9eec340e1bd4d74175dd392a2 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 43317d13dde547deabe618d1ea6110fa in current term 1: Already voted for candidate a204f0e9eec340e1bd4d74175dd392a2 in this term.
I20260924 16:13:08.886801 6073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7333ffb3d6eb4e5bab2d0e3bb5c92764" candidate_uuid: "43317d13dde547deabe618d1ea6110fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "284ce6491d364d2a83dad33520490442"
I20260924 16:13:08.888437 6140 leader_election.cc:302] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P 43317d13dde547deabe618d1ea6110fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 43317d13dde547deabe618d1ea6110fa; no voters: 284ce6491d364d2a83dad33520490442, a204f0e9eec340e1bd4d74175dd392a2
I20260924 16:13:08.888854 6437 raft_consensus.cc:2726] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P 43317d13dde547deabe618d1ea6110fa [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260924 16:13:08.888940 6208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7333ffb3d6eb4e5bab2d0e3bb5c92764" candidate_uuid: "a204f0e9eec340e1bd4d74175dd392a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43317d13dde547deabe618d1ea6110fa"
I20260924 16:13:08.889189 6208 raft_consensus.cc:2379] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P 43317d13dde547deabe618d1ea6110fa [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a204f0e9eec340e1bd4d74175dd392a2 in current term 1: Already voted for candidate 43317d13dde547deabe618d1ea6110fa in this term.
I20260924 16:13:08.889165 6440 leader_election.cc:288] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P a204f0e9eec340e1bd4d74175dd392a2 [CANDIDATE]: Term 1 election: Requested vote from peers 284ce6491d364d2a83dad33520490442 (127.29.234.1:41013), 43317d13dde547deabe618d1ea6110fa (127.29.234.2:36571)
I20260924 16:13:08.890173 6072 raft_consensus.cc:2454] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P 284ce6491d364d2a83dad33520490442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a204f0e9eec340e1bd4d74175dd392a2 in term 1.
I20260924 16:13:08.890544 6283 leader_election.cc:302] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P a204f0e9eec340e1bd4d74175dd392a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 284ce6491d364d2a83dad33520490442, a204f0e9eec340e1bd4d74175dd392a2; no voters: 43317d13dde547deabe618d1ea6110fa
I20260924 16:13:08.890791 6440 raft_consensus.cc:2781] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P a204f0e9eec340e1bd4d74175dd392a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:13:08.890957 6440 raft_consensus.cc:686] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P a204f0e9eec340e1bd4d74175dd392a2 [term 1 LEADER]: Becoming Leader. State: Replica: a204f0e9eec340e1bd4d74175dd392a2, State: Running, Role: LEADER
I20260924 16:13:08.891215 6440 consensus_queue.cc:237] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P a204f0e9eec340e1bd4d74175dd392a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284ce6491d364d2a83dad33520490442" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 41013 } } peers { permanent_uuid: "43317d13dde547deabe618d1ea6110fa" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36571 } } peers { permanent_uuid: "a204f0e9eec340e1bd4d74175dd392a2" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 33413 } }
I20260924 16:13:08.894169 5907 catalog_manager.cc:5495] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P a204f0e9eec340e1bd4d74175dd392a2 reported cstate change: term changed from 0 to 1, leader changed from <none> to a204f0e9eec340e1bd4d74175dd392a2 (127.29.234.3). New cstate: current_term: 1 leader_uuid: "a204f0e9eec340e1bd4d74175dd392a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284ce6491d364d2a83dad33520490442" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 41013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43317d13dde547deabe618d1ea6110fa" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a204f0e9eec340e1bd4d74175dd392a2" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 33413 } health_report { overall_health: HEALTHY } } }
Sep 24 16:13:08 dist-test-slave-v5j9 krb5kdc[5862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266385, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.234.62@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM
I20260924 16:13:08.996615 6469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:08.983704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:40365 (local address 127.0.0.1:49462)
0924 16:13:08.984499 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:08.984516 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:08.984643 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:08.984953 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:08.984956 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:08.985712 (+ 756us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:13:08.985917 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:08.985927 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:08.986967 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:08.986973 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:08.987977 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:08.987989 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:08.988656 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:08.988660 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:08.988788 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:08.989449 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:13:08.989467 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:13:08.991638 (+ 2171us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:13:08.994389 (+ 2751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:13:08.994401 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:13:08.994408 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:13:08.994831 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:13:08.995141 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:13:08.995144 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:13:08.995146 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:13:08.995300 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:13:08.995795 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:13:08.995802 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:13:08.995941 (+ 139us) client_negotiation.cc:769] Sending connection context
0924 16:13:08.996084 (+ 143us) client_negotiation.cc:240] Negotiation successful
0924 16:13:08.996323 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"thread_start_us":421,"threads_started":1}
I20260924 16:13:08.996803 6468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:08.983693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49462 (local address 127.29.234.62:40365)
0924 16:13:08.984411 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:08.984416 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:08.984598 (+ 182us) server_negotiation.cc:407] Connection header received
0924 16:13:08.984740 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:08.984744 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:08.984808 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:08.984909 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:08.986050 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:08.986831 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:08.988107 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:08.988513 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:08.988601 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:08.991816 (+ 3215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:13:08.991821 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:13:08.991824 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:13:08.991853 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:13:08.994250 (+ 2397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:08.994935 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:08.994938 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:08.994941 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:08.995047 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:08.995395 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:08.995398 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:08.995400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:08.995680 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:13:08.995787 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:08.996170 (+ 383us) server_negotiation.cc:299] Negotiation successful
0924 16:13:08.996457 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":99,"server-negotiator.queue_time_us":585,"thread_start_us":358,"threads_started":1}
I20260924 16:13:09.002293 5907 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.29.234.62@KRBTEST.COM'} at 127.0.0.1:49462:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 16:13:09.003474 5907 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:13:09.010234 6188 tablet_service.cc:1467] Processing CreateTablet for tablet 816a33a6a0aa426196d32ec8e51fb75d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=42368ee9e1084fdaad4711e76309fcf0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 16:13:09.010437 6331 tablet_service.cc:1467] Processing CreateTablet for tablet 816a33a6a0aa426196d32ec8e51fb75d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=42368ee9e1084fdaad4711e76309fcf0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 16:13:09.010633 6048 tablet_service.cc:1467] Processing CreateTablet for tablet 816a33a6a0aa426196d32ec8e51fb75d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=42368ee9e1084fdaad4711e76309fcf0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 16:13:09.010931 6188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 816a33a6a0aa426196d32ec8e51fb75d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:13:09.011009 6331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 816a33a6a0aa426196d32ec8e51fb75d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:13:09.011130 6048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 816a33a6a0aa426196d32ec8e51fb75d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:13:09.018841 6430 tablet_bootstrap.cc:492] T 816a33a6a0aa426196d32ec8e51fb75d P 43317d13dde547deabe618d1ea6110fa: Bootstrap starting.
I20260924 16:13:09.018967 6429 tablet_bootstrap.cc:492] T 816a33a6a0aa426196d32ec8e51fb75d P a204f0e9eec340e1bd4d74175dd392a2: Bootstrap starting.
I20260924 16:13:09.019138 6428 tablet_bootstrap.cc:492] T 816a33a6a0aa426196d32ec8e51fb75d P 284ce6491d364d2a83dad33520490442: Bootstrap starting.
I20260924 16:13:09.020287 6430 tablet_bootstrap.cc:654] T 816a33a6a0aa426196d32ec8e51fb75d P 43317d13dde547deabe618d1ea6110fa: Neither blocks nor log segments found. Creating new log.
I20260924 16:13:09.020746 6429 tablet_bootstrap.cc:654] T 816a33a6a0aa426196d32ec8e51fb75d P a204f0e9eec340e1bd4d74175dd392a2: Neither blocks nor log segments found. Creating new log.
I20260924 16:13:09.020829 6428 tablet_bootstrap.cc:654] T 816a33a6a0aa426196d32ec8e51fb75d P 284ce6491d364d2a83dad33520490442: Neither blocks nor log segments found. Creating new log.
I20260924 16:13:09.023041 6429 tablet_bootstrap.cc:492] T 816a33a6a0aa426196d32ec8e51fb75d P a204f0e9eec340e1bd4d74175dd392a2: No bootstrap required, opened a new log
I20260924 16:13:09.023206 6429 ts_tablet_manager.cc:1397] T 816a33a6a0aa426196d32ec8e51fb75d P a204f0e9eec340e1bd4d74175dd392a2: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.004s
I20260924 16:13:09.023427 6428 tablet_bootstrap.cc:492] T 816a33a6a0aa426196d32ec8e51fb75d P 284ce6491d364d2a83dad33520490442: No bootstrap required, opened a new log
I20260924 16:13:09.023567 6428 ts_tablet_manager.cc:1397] T 816a33a6a0aa426196d32ec8e51fb75d P 284ce6491d364d2a83dad33520490442: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20260924 16:13:09.024039 6430 tablet_bootstrap.cc:492] T 816a33a6a0aa426196d32ec8e51fb75d P 43317d13dde547deabe618d1ea6110fa: No bootstrap required, opened a new log
I20260924 16:13:09.023991 6429 raft_consensus.cc:348] T 816a33a6a0aa426196d32ec8e51fb75d P a204f0e9eec340e1bd4d74175dd392a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43317d13dde547deabe618d1ea6110fa" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36571 } } peers { permanent_uuid: "a204f0e9eec340e1bd4d74175dd392a2" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 33413 } } peers { permanent_uuid: "284ce6491d364d2a83dad33520490442" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 41013 } }
I20260924 16:13:09.024196 6430 ts_tablet_manager.cc:1397] T 816a33a6a0aa426196d32ec8e51fb75d P 43317d13dde547deabe618d1ea6110fa: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 16:13:09.024276 6429 raft_consensus.cc:374] T 816a33a6a0aa426196d32ec8e51fb75d P a204f0e9eec340e1bd4d74175dd392a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:13:09.024367 6429 raft_consensus.cc:729] T 816a33a6a0aa426196d32ec8e51fb75d P a204f0e9eec340e1bd4d74175dd392a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a204f0e9eec340e1bd4d74175dd392a2, State: Initialized, Role: FOLLOWER
I20260924 16:13:09.024370 6428 raft_consensus.cc:348] T 816a33a6a0aa426196d32ec8e51fb75d P 284ce6491d364d2a83dad33520490442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43317d13dde547deabe618d1ea6110fa" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36571 } } peers { permanent_uuid: "a204f0e9eec340e1bd4d74175dd392a2" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 33413 } } peers { permanent_uuid: "284ce6491d364d2a83dad33520490442" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 41013 } }
I20260924 16:13:09.024632 6428 raft_consensus.cc:374] T 816a33a6a0aa426196d32ec8e51fb75d P 284ce6491d364d2a83dad33520490442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:13:09.024715 6428 raft_consensus.cc:729] T 816a33a6a0aa426196d32ec8e51fb75d P 284ce6491d364d2a83dad33520490442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 284ce6491d364d2a83dad33520490442, State: Initialized, Role: FOLLOWER
I20260924 16:13:09.024605 6429 consensus_queue.cc:260] T 816a33a6a0aa426196d32ec8e51fb75d P a204f0e9eec340e1bd4d74175dd392a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43317d13dde547deabe618d1ea6110fa" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36571 } } peers { permanent_uuid: "a204f0e9eec340e1bd4d74175dd392a2" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 33413 } } peers { permanent_uuid: "284ce6491d364d2a83dad33520490442" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 41013 } }
I20260924 16:13:09.024952 6428 consensus_queue.cc:260] T 816a33a6a0aa426196d32ec8e51fb75d P 284ce6491d364d2a83dad33520490442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43317d13dde547deabe618d1ea6110fa" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36571 } } peers { permanent_uuid: "a204f0e9eec340e1bd4d74175dd392a2" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 33413 } } peers { permanent_uuid: "284ce6491d364d2a83dad33520490442" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 41013 } }
I20260924 16:13:09.025151 6429 ts_tablet_manager.cc:1428] T 816a33a6a0aa426196d32ec8e51fb75d P a204f0e9eec340e1bd4d74175dd392a2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260924 16:13:09.025065 6430 raft_consensus.cc:348] T 816a33a6a0aa426196d32ec8e51fb75d P 43317d13dde547deabe618d1ea6110fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43317d13dde547deabe618d1ea6110fa" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36571 } } peers { permanent_uuid: "a204f0e9eec340e1bd4d74175dd392a2" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 33413 } } peers { permanent_uuid: "284ce6491d364d2a83dad33520490442" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 41013 } }
I20260924 16:13:09.025622 6428 ts_tablet_manager.cc:1428] T 816a33a6a0aa426196d32ec8e51fb75d P 284ce6491d364d2a83dad33520490442: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 16:13:09.025511 6440 tablet_replica.cc:439] T 816a33a6a0aa426196d32ec8e51fb75d P a204f0e9eec340e1bd4d74175dd392a2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43317d13dde547deabe618d1ea6110fa" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36571 } } peers { permanent_uuid: "a204f0e9eec340e1bd4d74175dd392a2" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 33413 } } peers { permanent_uuid: "284ce6491d364d2a83dad33520490442" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 41013 } } }
I20260924 16:13:09.025777 6440 tablet_replica.cc:442] T 816a33a6a0aa426196d32ec8e51fb75d P a204f0e9eec340e1bd4d74175dd392a2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:13:09.025516 6435 tablet_replica.cc:439] T 816a33a6a0aa426196d32ec8e51fb75d P 284ce6491d364d2a83dad33520490442: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43317d13dde547deabe618d1ea6110fa" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36571 } } peers { permanent_uuid: "a204f0e9eec340e1bd4d74175dd392a2" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 33413 } } peers { permanent_uuid: "284ce6491d364d2a83dad33520490442" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 41013 } } }
I20260924 16:13:09.025888 6435 tablet_replica.cc:442] T 816a33a6a0aa426196d32ec8e51fb75d P 284ce6491d364d2a83dad33520490442: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:13:09.026364 6430 raft_consensus.cc:374] T 816a33a6a0aa426196d32ec8e51fb75d P 43317d13dde547deabe618d1ea6110fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:13:09.026470 6430 raft_consensus.cc:729] T 816a33a6a0aa426196d32ec8e51fb75d P 43317d13dde547deabe618d1ea6110fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43317d13dde547deabe618d1ea6110fa, State: Initialized, Role: FOLLOWER
I20260924 16:13:09.026707 6430 consensus_queue.cc:260] T 816a33a6a0aa426196d32ec8e51fb75d P 43317d13dde547deabe618d1ea6110fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43317d13dde547deabe618d1ea6110fa" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36571 } } peers { permanent_uuid: "a204f0e9eec340e1bd4d74175dd392a2" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 33413 } } peers { permanent_uuid: "284ce6491d364d2a83dad33520490442" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 41013 } }
I20260924 16:13:09.027148 6437 tablet_replica.cc:439] T 816a33a6a0aa426196d32ec8e51fb75d P 43317d13dde547deabe618d1ea6110fa: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43317d13dde547deabe618d1ea6110fa" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36571 } } peers { permanent_uuid: "a204f0e9eec340e1bd4d74175dd392a2" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 33413 } } peers { permanent_uuid: "284ce6491d364d2a83dad33520490442" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 41013 } } }
I20260924 16:13:09.027459 6437 tablet_replica.cc:442] T 816a33a6a0aa426196d32ec8e51fb75d P 43317d13dde547deabe618d1ea6110fa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:13:09.028059 6430 ts_tablet_manager.cc:1428] T 816a33a6a0aa426196d32ec8e51fb75d P 43317d13dde547deabe618d1ea6110fa: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260924 16:13:09.168179 6440 raft_consensus.cc:482] T 816a33a6a0aa426196d32ec8e51fb75d P a204f0e9eec340e1bd4d74175dd392a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:13:09.168437 6440 raft_consensus.cc:504] T 816a33a6a0aa426196d32ec8e51fb75d P a204f0e9eec340e1bd4d74175dd392a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43317d13dde547deabe618d1ea6110fa" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36571 } } peers { permanent_uuid: "a204f0e9eec340e1bd4d74175dd392a2" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 33413 } } peers { permanent_uuid: "284ce6491d364d2a83dad33520490442" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 41013 } }
I20260924 16:13:09.169175 6440 leader_election.cc:288] T 816a33a6a0aa426196d32ec8e51fb75d P a204f0e9eec340e1bd4d74175dd392a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43317d13dde547deabe618d1ea6110fa (127.29.234.2:36571), 284ce6491d364d2a83dad33520490442 (127.29.234.1:41013)
I20260924 16:13:09.169757 6208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "816a33a6a0aa426196d32ec8e51fb75d" candidate_uuid: "a204f0e9eec340e1bd4d74175dd392a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43317d13dde547deabe618d1ea6110fa" is_pre_election: true
I20260924 16:13:09.169740 6072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "816a33a6a0aa426196d32ec8e51fb75d" candidate_uuid: "a204f0e9eec340e1bd4d74175dd392a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "284ce6491d364d2a83dad33520490442" is_pre_election: true
I20260924 16:13:09.170010 6208 raft_consensus.cc:2454] T 816a33a6a0aa426196d32ec8e51fb75d P 43317d13dde547deabe618d1ea6110fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a204f0e9eec340e1bd4d74175dd392a2 in term 0.
I20260924 16:13:09.170028 6072 raft_consensus.cc:2454] T 816a33a6a0aa426196d32ec8e51fb75d P 284ce6491d364d2a83dad33520490442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a204f0e9eec340e1bd4d74175dd392a2 in term 0.
I20260924 16:13:09.170454 6284 leader_election.cc:302] T 816a33a6a0aa426196d32ec8e51fb75d P a204f0e9eec340e1bd4d74175dd392a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43317d13dde547deabe618d1ea6110fa, a204f0e9eec340e1bd4d74175dd392a2; no voters:
I20260924 16:13:09.170783 6440 raft_consensus.cc:2781] T 816a33a6a0aa426196d32ec8e51fb75d P a204f0e9eec340e1bd4d74175dd392a2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:13:09.170892 6440 raft_consensus.cc:482] T 816a33a6a0aa426196d32ec8e51fb75d P a204f0e9eec340e1bd4d74175dd392a2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:13:09.171030 6440 raft_consensus.cc:3037] T 816a33a6a0aa426196d32ec8e51fb75d P a204f0e9eec340e1bd4d74175dd392a2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:13:09.173166 6440 raft_consensus.cc:504] T 816a33a6a0aa426196d32ec8e51fb75d P a204f0e9eec340e1bd4d74175dd392a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43317d13dde547deabe618d1ea6110fa" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36571 } } peers { permanent_uuid: "a204f0e9eec340e1bd4d74175dd392a2" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 33413 } } peers { permanent_uuid: "284ce6491d364d2a83dad33520490442" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 41013 } }
I20260924 16:13:09.173749 6440 leader_election.cc:288] T 816a33a6a0aa426196d32ec8e51fb75d P a204f0e9eec340e1bd4d74175dd392a2 [CANDIDATE]: Term 1 election: Requested vote from peers 43317d13dde547deabe618d1ea6110fa (127.29.234.2:36571), 284ce6491d364d2a83dad33520490442 (127.29.234.1:41013)
I20260924 16:13:09.174204 6072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "816a33a6a0aa426196d32ec8e51fb75d" candidate_uuid: "a204f0e9eec340e1bd4d74175dd392a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "284ce6491d364d2a83dad33520490442"
I20260924 16:13:09.174213 6208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "816a33a6a0aa426196d32ec8e51fb75d" candidate_uuid: "a204f0e9eec340e1bd4d74175dd392a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43317d13dde547deabe618d1ea6110fa"
I20260924 16:13:09.174424 6208 raft_consensus.cc:3037] T 816a33a6a0aa426196d32ec8e51fb75d P 43317d13dde547deabe618d1ea6110fa [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:13:09.174417 6072 raft_consensus.cc:3037] T 816a33a6a0aa426196d32ec8e51fb75d P 284ce6491d364d2a83dad33520490442 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:13:09.176383 6072 raft_consensus.cc:2454] T 816a33a6a0aa426196d32ec8e51fb75d P 284ce6491d364d2a83dad33520490442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a204f0e9eec340e1bd4d74175dd392a2 in term 1.
I20260924 16:13:09.176396 6208 raft_consensus.cc:2454] T 816a33a6a0aa426196d32ec8e51fb75d P 43317d13dde547deabe618d1ea6110fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a204f0e9eec340e1bd4d74175dd392a2 in term 1.
I20260924 16:13:09.176836 6283 leader_election.cc:302] T 816a33a6a0aa426196d32ec8e51fb75d P a204f0e9eec340e1bd4d74175dd392a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 284ce6491d364d2a83dad33520490442, a204f0e9eec340e1bd4d74175dd392a2; no voters:
I20260924 16:13:09.177130 6440 raft_consensus.cc:2781] T 816a33a6a0aa426196d32ec8e51fb75d P a204f0e9eec340e1bd4d74175dd392a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:13:09.177278 6440 raft_consensus.cc:686] T 816a33a6a0aa426196d32ec8e51fb75d P a204f0e9eec340e1bd4d74175dd392a2 [term 1 LEADER]: Becoming Leader. State: Replica: a204f0e9eec340e1bd4d74175dd392a2, State: Running, Role: LEADER
I20260924 16:13:09.177585 6440 consensus_queue.cc:237] T 816a33a6a0aa426196d32ec8e51fb75d P a204f0e9eec340e1bd4d74175dd392a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43317d13dde547deabe618d1ea6110fa" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36571 } } peers { permanent_uuid: "a204f0e9eec340e1bd4d74175dd392a2" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 33413 } } peers { permanent_uuid: "284ce6491d364d2a83dad33520490442" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 41013 } }
I20260924 16:13:09.179143 6436 tablet_replica.cc:439] T 816a33a6a0aa426196d32ec8e51fb75d P a204f0e9eec340e1bd4d74175dd392a2: TxnStatusTablet state changed. Reason: New leader a204f0e9eec340e1bd4d74175dd392a2. Latest consensus state: current_term: 1 leader_uuid: "a204f0e9eec340e1bd4d74175dd392a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43317d13dde547deabe618d1ea6110fa" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36571 } } peers { permanent_uuid: "a204f0e9eec340e1bd4d74175dd392a2" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 33413 } } peers { permanent_uuid: "284ce6491d364d2a83dad33520490442" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 41013 } } }
I20260924 16:13:09.179386 6436 tablet_replica.cc:442] T 816a33a6a0aa426196d32ec8e51fb75d P a204f0e9eec340e1bd4d74175dd392a2: This TxnStatusTablet replica's current role is: LEADER
I20260924 16:13:09.179953 6474 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 16:13:09.180207 6474 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 16:13:09.180634 5907 catalog_manager.cc:5495] T 816a33a6a0aa426196d32ec8e51fb75d P a204f0e9eec340e1bd4d74175dd392a2 reported cstate change: term changed from 0 to 1, leader changed from <none> to a204f0e9eec340e1bd4d74175dd392a2 (127.29.234.3). New cstate: current_term: 1 leader_uuid: "a204f0e9eec340e1bd4d74175dd392a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43317d13dde547deabe618d1ea6110fa" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a204f0e9eec340e1bd4d74175dd392a2" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 33413 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "284ce6491d364d2a83dad33520490442" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 41013 } health_report { overall_health: UNKNOWN } } }
I20260924 16:13:09.220319 6469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:09.214187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.3:33413 (local address 127.0.0.1:50430)
0924 16:13:09.214480 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:09.214496 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:09.214676 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:09.215166 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:09.215170 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:09.215184 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 16:13:09.215419 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:09.215431 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:09.216596 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:09.216601 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:09.218161 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:09.218183 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:09.219013 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:09.219019 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:09.219203 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:09.219269 (+ 66us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 16:13:09.219684 (+ 415us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 16:13:09.219720 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 16:13:09.219828 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 16:13:09.219989 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102}
I20260924 16:13:09.220328 6424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:09.214162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50430 (local address 127.29.234.3:33413)
0924 16:13:09.214372 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:09.214379 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:09.214571 (+ 192us) server_negotiation.cc:407] Connection header received
0924 16:13:09.214943 (+ 372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:09.214948 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:09.215030 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:09.215135 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:13:09.215571 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:09.216451 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:09.218376 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:09.218846 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:09.218999 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:09.219461 (+ 462us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:13:09.219574 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:13:09.219683 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:09.219870 (+ 187us) server_negotiation.cc:299] Negotiation successful
0924 16:13:09.219953 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 16:13:09.224918 6208 raft_consensus.cc:1261] T 816a33a6a0aa426196d32ec8e51fb75d P 43317d13dde547deabe618d1ea6110fa [term 1 FOLLOWER]: Refusing update from remote peer a204f0e9eec340e1bd4d74175dd392a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:13:09.224946 6072 raft_consensus.cc:1261] T 816a33a6a0aa426196d32ec8e51fb75d P 284ce6491d364d2a83dad33520490442 [term 1 FOLLOWER]: Refusing update from remote peer a204f0e9eec340e1bd4d74175dd392a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:13:09.225674 6436 consensus_queue.cc:1035] T 816a33a6a0aa426196d32ec8e51fb75d P a204f0e9eec340e1bd4d74175dd392a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "284ce6491d364d2a83dad33520490442" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 41013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:13:09.226032 6440 consensus_queue.cc:1035] T 816a33a6a0aa426196d32ec8e51fb75d P a204f0e9eec340e1bd4d74175dd392a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43317d13dde547deabe618d1ea6110fa" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:13:09.234449 6437 tablet_replica.cc:439] T 816a33a6a0aa426196d32ec8e51fb75d P 43317d13dde547deabe618d1ea6110fa: TxnStatusTablet state changed. Reason: New leader a204f0e9eec340e1bd4d74175dd392a2. Latest consensus state: current_term: 1 leader_uuid: "a204f0e9eec340e1bd4d74175dd392a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43317d13dde547deabe618d1ea6110fa" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36571 } } peers { permanent_uuid: "a204f0e9eec340e1bd4d74175dd392a2" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 33413 } } peers { permanent_uuid: "284ce6491d364d2a83dad33520490442" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 41013 } } }
I20260924 16:13:09.234897 6437 tablet_replica.cc:442] T 816a33a6a0aa426196d32ec8e51fb75d P 43317d13dde547deabe618d1ea6110fa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:13:09.236856 6435 tablet_replica.cc:439] T 816a33a6a0aa426196d32ec8e51fb75d P 284ce6491d364d2a83dad33520490442: TxnStatusTablet state changed. Reason: New leader a204f0e9eec340e1bd4d74175dd392a2. Latest consensus state: current_term: 1 leader_uuid: "a204f0e9eec340e1bd4d74175dd392a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43317d13dde547deabe618d1ea6110fa" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36571 } } peers { permanent_uuid: "a204f0e9eec340e1bd4d74175dd392a2" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 33413 } } peers { permanent_uuid: "284ce6491d364d2a83dad33520490442" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 41013 } } }
I20260924 16:13:09.237191 6435 tablet_replica.cc:442] T 816a33a6a0aa426196d32ec8e51fb75d P 284ce6491d364d2a83dad33520490442: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:13:09.239051 6477 mvcc.cc:205] Tried to move back new op lower bound from 7332931130258874368 to 7332931130073780224. Current Snapshot: MvccSnapshot[applied={T|T < 7332931130258874368}]
I20260924 16:13:09.239876 6478 mvcc.cc:205] Tried to move back new op lower bound from 7332931130258874368 to 7332931130073780224. Current Snapshot: MvccSnapshot[applied={T|T < 7332931130258874368}]
I20260924 16:13:09.245942 6437 tablet_replica.cc:439] T 816a33a6a0aa426196d32ec8e51fb75d P 43317d13dde547deabe618d1ea6110fa: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a204f0e9eec340e1bd4d74175dd392a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43317d13dde547deabe618d1ea6110fa" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36571 } } peers { permanent_uuid: "a204f0e9eec340e1bd4d74175dd392a2" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 33413 } } peers { permanent_uuid: "284ce6491d364d2a83dad33520490442" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 41013 } } }
I20260924 16:13:09.246320 6437 tablet_replica.cc:442] T 816a33a6a0aa426196d32ec8e51fb75d P 43317d13dde547deabe618d1ea6110fa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:13:09.248685 6435 tablet_replica.cc:439] T 816a33a6a0aa426196d32ec8e51fb75d P 284ce6491d364d2a83dad33520490442: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a204f0e9eec340e1bd4d74175dd392a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43317d13dde547deabe618d1ea6110fa" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36571 } } peers { permanent_uuid: "a204f0e9eec340e1bd4d74175dd392a2" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 33413 } } peers { permanent_uuid: "284ce6491d364d2a83dad33520490442" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 41013 } } }
I20260924 16:13:09.248961 6435 tablet_replica.cc:442] T 816a33a6a0aa426196d32ec8e51fb75d P 284ce6491d364d2a83dad33520490442: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 16:13:09.250072 6436 tablet_replica.cc:439] T 816a33a6a0aa426196d32ec8e51fb75d P a204f0e9eec340e1bd4d74175dd392a2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a204f0e9eec340e1bd4d74175dd392a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43317d13dde547deabe618d1ea6110fa" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36571 } } peers { permanent_uuid: "a204f0e9eec340e1bd4d74175dd392a2" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 33413 } } peers { permanent_uuid: "284ce6491d364d2a83dad33520490442" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 41013 } } }
I20260924 16:13:09.250496 6436 tablet_replica.cc:442] T 816a33a6a0aa426196d32ec8e51fb75d P a204f0e9eec340e1bd4d74175dd392a2: This TxnStatusTablet replica's current role is: LEADER
I20260924 16:13:09.250761 6436 tablet_replica.cc:439] T 816a33a6a0aa426196d32ec8e51fb75d P a204f0e9eec340e1bd4d74175dd392a2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a204f0e9eec340e1bd4d74175dd392a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43317d13dde547deabe618d1ea6110fa" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36571 } } peers { permanent_uuid: "a204f0e9eec340e1bd4d74175dd392a2" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 33413 } } peers { permanent_uuid: "284ce6491d364d2a83dad33520490442" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 41013 } } }
I20260924 16:13:09.252931 6436 tablet_replica.cc:442] T 816a33a6a0aa426196d32ec8e51fb75d P a204f0e9eec340e1bd4d74175dd392a2: This TxnStatusTablet replica's current role is: LEADER
I20260924 16:13:09.272926 6424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:09.266118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.3:50917 (local address 127.29.234.3:33413)
0924 16:13:09.266353 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:09.266362 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:09.267014 (+ 652us) server_negotiation.cc:407] Connection header received
0924 16:13:09.267196 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:09.267199 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:09.267270 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:09.267348 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:13:09.267932 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:09.268711 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:09.270890 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:09.271377 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:09.271521 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:09.271940 (+ 419us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:13:09.272076 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:13:09.272174 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:09.272378 (+ 204us) server_negotiation.cc:299] Negotiation successful
0924 16:13:09.272548 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":66}
I20260924 16:13:09.273032 6487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:09.266124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.3:33413 (local address 127.29.234.3:50917)
0924 16:13:09.266913 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:09.266932 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:09.267076 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:09.267383 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:09.267388 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:09.267401 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 16:13:09.267780 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:09.267792 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:09.268833 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:09.268839 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:09.270715 (+ 1876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:09.270730 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:09.271552 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:09.271559 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:09.271713 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:09.271769 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 16:13:09.272174 (+ 405us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 16:13:09.272186 (+ 12us) client_negotiation.cc:769] Sending connection context
0924 16:13:09.272363 (+ 177us) client_negotiation.cc:240] Negotiation successful
0924 16:13:09.272586 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"mutex_wait_us":42,"thread_start_us":381,"threads_started":1}
I20260924 16:13:09.282940 6208 raft_consensus.cc:1261] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P 43317d13dde547deabe618d1ea6110fa [term 1 FOLLOWER]: Refusing update from remote peer a204f0e9eec340e1bd4d74175dd392a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:13:09.283185 6072 raft_consensus.cc:1261] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P 284ce6491d364d2a83dad33520490442 [term 1 FOLLOWER]: Refusing update from remote peer a204f0e9eec340e1bd4d74175dd392a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:13:09.283839 6440 consensus_queue.cc:1035] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P a204f0e9eec340e1bd4d74175dd392a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43317d13dde547deabe618d1ea6110fa" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:13:09.284089 6436 consensus_queue.cc:1035] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P a204f0e9eec340e1bd4d74175dd392a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "284ce6491d364d2a83dad33520490442" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 41013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:13:09.344384 5907 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49458:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:13:09.344631 5907 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49458:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:13:09.347551 5907 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 441441046a974081ab6e0e4a1576c8f7: Sending DeleteTablet for 3 replicas of tablet 7333ffb3d6eb4e5bab2d0e3bb5c92764
I20260924 16:13:09.348296 6048 tablet_service.cc:1514] Processing DeleteTablet for tablet 7333ffb3d6eb4e5bab2d0e3bb5c92764 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:13:09 UTC) from {username='oryx', principal='oryx/127.29.234.62@KRBTEST.COM'} at 127.0.0.1:49384
I20260924 16:13:09.348395 6188 tablet_service.cc:1514] Processing DeleteTablet for tablet 7333ffb3d6eb4e5bab2d0e3bb5c92764 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:13:09 UTC) from {username='oryx', principal='oryx/127.29.234.62@KRBTEST.COM'} at 127.0.0.1:56866
I20260924 16:13:09.348451 6330 tablet_service.cc:1514] Processing DeleteTablet for tablet 7333ffb3d6eb4e5bab2d0e3bb5c92764 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:13:09 UTC) from {username='oryx', principal='oryx/127.29.234.62@KRBTEST.COM'} at 127.0.0.1:50420
I20260924 16:13:09.359755 6492 tablet_replica.cc:330] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P 284ce6491d364d2a83dad33520490442: stopping tablet replica
I20260924 16:13:09.359758 6494 tablet_replica.cc:330] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P a204f0e9eec340e1bd4d74175dd392a2: stopping tablet replica
I20260924 16:13:09.359972 6494 raft_consensus.cc:2229] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P a204f0e9eec340e1bd4d74175dd392a2 [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:13:09.360039 6492 raft_consensus.cc:2229] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P 284ce6491d364d2a83dad33520490442 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:13:09.360391 6492 raft_consensus.cc:2258] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P 284ce6491d364d2a83dad33520490442 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:13:09.360419 6494 raft_consensus.cc:2258] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P a204f0e9eec340e1bd4d74175dd392a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:13:09.361537 6494 ts_tablet_manager.cc:1905] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P a204f0e9eec340e1bd4d74175dd392a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:13:09.361531 6492 ts_tablet_manager.cc:1905] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P 284ce6491d364d2a83dad33520490442: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:13:09.368613 6492 ts_tablet_manager.cc:1918] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P 284ce6491d364d2a83dad33520490442: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 16:13:09.368700 6494 ts_tablet_manager.cc:1918] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P a204f0e9eec340e1bd4d74175dd392a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 16:13:09.368942 6494 log.cc:1196] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P a204f0e9eec340e1bd4d74175dd392a2: Deleting WAL directory at /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/minicluster-data/ts-2/wal/wals/7333ffb3d6eb4e5bab2d0e3bb5c92764
I20260924 16:13:09.369226 6493 tablet_replica.cc:330] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P 43317d13dde547deabe618d1ea6110fa: stopping tablet replica
I20260924 16:13:09.369483 6493 raft_consensus.cc:2229] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P 43317d13dde547deabe618d1ea6110fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:13:09.369668 6494 ts_tablet_manager.cc:1939] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P a204f0e9eec340e1bd4d74175dd392a2: Deleting consensus metadata
I20260924 16:13:09.369827 6493 raft_consensus.cc:2258] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P 43317d13dde547deabe618d1ea6110fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:13:09.370965 6493 ts_tablet_manager.cc:1905] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P 43317d13dde547deabe618d1ea6110fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:13:09.371019 5891 catalog_manager.cc:4841] TS a204f0e9eec340e1bd4d74175dd392a2 (127.29.234.3:33413): tablet 7333ffb3d6eb4e5bab2d0e3bb5c92764 (table test-table [id=f9fc73d97bcf4087ae0d8787d5ec6c7e]) successfully deleted
I20260924 16:13:09.371572 6492 log.cc:1196] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P 284ce6491d364d2a83dad33520490442: Deleting WAL directory at /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/minicluster-data/ts-0/wal/wals/7333ffb3d6eb4e5bab2d0e3bb5c92764
I20260924 16:13:09.372102 6492 ts_tablet_manager.cc:1939] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P 284ce6491d364d2a83dad33520490442: Deleting consensus metadata
I20260924 16:13:09.373172 5892 catalog_manager.cc:4841] TS 284ce6491d364d2a83dad33520490442 (127.29.234.1:41013): tablet 7333ffb3d6eb4e5bab2d0e3bb5c92764 (table test-table [id=f9fc73d97bcf4087ae0d8787d5ec6c7e]) successfully deleted
I20260924 16:13:09.380373 6493 ts_tablet_manager.cc:1918] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P 43317d13dde547deabe618d1ea6110fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 16:13:09.380654 6493 log.cc:1196] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P 43317d13dde547deabe618d1ea6110fa: Deleting WAL directory at /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/minicluster-data/ts-1/wal/wals/7333ffb3d6eb4e5bab2d0e3bb5c92764
I20260924 16:13:09.381135 6493 ts_tablet_manager.cc:1939] T 7333ffb3d6eb4e5bab2d0e3bb5c92764 P 43317d13dde547deabe618d1ea6110fa: Deleting consensus metadata
I20260924 16:13:09.382239 5893 catalog_manager.cc:4841] TS 43317d13dde547deabe618d1ea6110fa (127.29.234.2:36571): tablet 7333ffb3d6eb4e5bab2d0e3bb5c92764 (table test-table [id=f9fc73d97bcf4087ae0d8787d5ec6c7e]) successfully deleted
Sep 24 16:13:09 dist-test-slave-v5j9 krb5kdc[5862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 16:13:09 dist-test-slave-v5j9 krb5kdc[5862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:13:09 dist-test-slave-v5j9 krb5kdc[5862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 16:13:09.423756 6468 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:13:09.412443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49468 (local address 127.29.234.62:40365)
0924 16:13:09.412692 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:09.412700 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:09.413081 (+ 381us) server_negotiation.cc:407] Connection header received
0924 16:13:09.413247 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:09.413261 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:09.413371 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:09.413495 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:09.414971 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:09.415766 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:09.417043 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:09.417565 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:09.417724 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:09.423494 (+ 5770us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49468: BlockingRecv error: recv got EOF from 127.0.0.1:49468 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 24 16:13:09 dist-test-slave-v5j9 krb5kdc[5862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM
I20260924 16:13:09.446645 6468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:09.430659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49472 (local address 127.29.234.62:40365)
0924 16:13:09.431401 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:09.431406 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:09.431826 (+ 420us) server_negotiation.cc:407] Connection header received
0924 16:13:09.432030 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:09.432036 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:09.432121 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:09.432231 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:09.433449 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:09.434587 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:09.435810 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:09.436443 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:09.436568 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:09.440531 (+ 3963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:13:09.440538 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:13:09.440543 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:13:09.440587 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:13:09.443941 (+ 3354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:09.444637 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:09.444643 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:09.444647 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:09.444806 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:09.445155 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:09.445161 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:09.445164 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:09.445549 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:13:09.445816 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:09.446013 (+ 197us) server_negotiation.cc:299] Negotiation successful
0924 16:13:09.446272 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626}
I20260924 16:13:09.450151 5907 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49472:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:13:09.450762 5907 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:13:09.457355 6048 tablet_service.cc:1467] Processing CreateTablet for tablet c45fde87756a4a2ea51423eef9aa814a (DEFAULT_TABLE table=test-table [id=f50c33298c4f4cb4b87f1eb863f6575c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:13:09.457484 6188 tablet_service.cc:1467] Processing CreateTablet for tablet c45fde87756a4a2ea51423eef9aa814a (DEFAULT_TABLE table=test-table [id=f50c33298c4f4cb4b87f1eb863f6575c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:13:09.457823 6048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c45fde87756a4a2ea51423eef9aa814a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:13:09.457916 6188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c45fde87756a4a2ea51423eef9aa814a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:13:09.462424 6428 tablet_bootstrap.cc:492] T c45fde87756a4a2ea51423eef9aa814a P 284ce6491d364d2a83dad33520490442: Bootstrap starting.
I20260924 16:13:09.463099 6330 tablet_service.cc:1467] Processing CreateTablet for tablet c45fde87756a4a2ea51423eef9aa814a (DEFAULT_TABLE table=test-table [id=f50c33298c4f4cb4b87f1eb863f6575c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:13:09.463546 6330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c45fde87756a4a2ea51423eef9aa814a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:13:09.464023 6428 tablet_bootstrap.cc:654] T c45fde87756a4a2ea51423eef9aa814a P 284ce6491d364d2a83dad33520490442: Neither blocks nor log segments found. Creating new log.
I20260924 16:13:09.464268 6430 tablet_bootstrap.cc:492] T c45fde87756a4a2ea51423eef9aa814a P 43317d13dde547deabe618d1ea6110fa: Bootstrap starting.
I20260924 16:13:09.466015 6430 tablet_bootstrap.cc:654] T c45fde87756a4a2ea51423eef9aa814a P 43317d13dde547deabe618d1ea6110fa: Neither blocks nor log segments found. Creating new log.
I20260924 16:13:09.475365 6429 tablet_bootstrap.cc:492] T c45fde87756a4a2ea51423eef9aa814a P a204f0e9eec340e1bd4d74175dd392a2: Bootstrap starting.
I20260924 16:13:09.477368 6429 tablet_bootstrap.cc:654] T c45fde87756a4a2ea51423eef9aa814a P a204f0e9eec340e1bd4d74175dd392a2: Neither blocks nor log segments found. Creating new log.
I20260924 16:13:09.478750 6428 tablet_bootstrap.cc:492] T c45fde87756a4a2ea51423eef9aa814a P 284ce6491d364d2a83dad33520490442: No bootstrap required, opened a new log
I20260924 16:13:09.478986 6428 ts_tablet_manager.cc:1397] T c45fde87756a4a2ea51423eef9aa814a P 284ce6491d364d2a83dad33520490442: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.007s
I20260924 16:13:09.479861 6428 raft_consensus.cc:348] T c45fde87756a4a2ea51423eef9aa814a P 284ce6491d364d2a83dad33520490442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284ce6491d364d2a83dad33520490442" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 41013 } } peers { permanent_uuid: "43317d13dde547deabe618d1ea6110fa" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36571 } } peers { permanent_uuid: "a204f0e9eec340e1bd4d74175dd392a2" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 33413 } }
I20260924 16:13:09.480408 6428 raft_consensus.cc:374] T c45fde87756a4a2ea51423eef9aa814a P 284ce6491d364d2a83dad33520490442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:13:09.480557 6428 raft_consensus.cc:729] T c45fde87756a4a2ea51423eef9aa814a P 284ce6491d364d2a83dad33520490442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 284ce6491d364d2a83dad33520490442, State: Initialized, Role: FOLLOWER
I20260924 16:13:09.481060 6430 tablet_bootstrap.cc:492] T c45fde87756a4a2ea51423eef9aa814a P 43317d13dde547deabe618d1ea6110fa: No bootstrap required, opened a new log
I20260924 16:13:09.480907 6428 consensus_queue.cc:260] T c45fde87756a4a2ea51423eef9aa814a P 284ce6491d364d2a83dad33520490442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284ce6491d364d2a83dad33520490442" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 41013 } } peers { permanent_uuid: "43317d13dde547deabe618d1ea6110fa" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36571 } } peers { permanent_uuid: "a204f0e9eec340e1bd4d74175dd392a2" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 33413 } }
I20260924 16:13:09.481225 6430 ts_tablet_manager.cc:1397] T c45fde87756a4a2ea51423eef9aa814a P 43317d13dde547deabe618d1ea6110fa: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.004s
I20260924 16:13:09.482053 6430 raft_consensus.cc:348] T c45fde87756a4a2ea51423eef9aa814a P 43317d13dde547deabe618d1ea6110fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284ce6491d364d2a83dad33520490442" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 41013 } } peers { permanent_uuid: "43317d13dde547deabe618d1ea6110fa" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36571 } } peers { permanent_uuid: "a204f0e9eec340e1bd4d74175dd392a2" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 33413 } }
I20260924 16:13:09.482321 6430 raft_consensus.cc:374] T c45fde87756a4a2ea51423eef9aa814a P 43317d13dde547deabe618d1ea6110fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:13:09.482424 6430 raft_consensus.cc:729] T c45fde87756a4a2ea51423eef9aa814a P 43317d13dde547deabe618d1ea6110fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43317d13dde547deabe618d1ea6110fa, State: Initialized, Role: FOLLOWER
I20260924 16:13:09.482647 6430 consensus_queue.cc:260] T c45fde87756a4a2ea51423eef9aa814a P 43317d13dde547deabe618d1ea6110fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284ce6491d364d2a83dad33520490442" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 41013 } } peers { permanent_uuid: "43317d13dde547deabe618d1ea6110fa" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36571 } } peers { permanent_uuid: "a204f0e9eec340e1bd4d74175dd392a2" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 33413 } }
I20260924 16:13:09.483274 6430 ts_tablet_manager.cc:1428] T c45fde87756a4a2ea51423eef9aa814a P 43317d13dde547deabe618d1ea6110fa: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 16:13:09.485901 6428 ts_tablet_manager.cc:1428] T c45fde87756a4a2ea51423eef9aa814a P 284ce6491d364d2a83dad33520490442: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260924 16:13:09.485927 6429 tablet_bootstrap.cc:492] T c45fde87756a4a2ea51423eef9aa814a P a204f0e9eec340e1bd4d74175dd392a2: No bootstrap required, opened a new log
I20260924 16:13:09.486109 6429 ts_tablet_manager.cc:1397] T c45fde87756a4a2ea51423eef9aa814a P a204f0e9eec340e1bd4d74175dd392a2: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.008s
I20260924 16:13:09.486927 6429 raft_consensus.cc:348] T c45fde87756a4a2ea51423eef9aa814a P a204f0e9eec340e1bd4d74175dd392a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284ce6491d364d2a83dad33520490442" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 41013 } } peers { permanent_uuid: "43317d13dde547deabe618d1ea6110fa" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36571 } } peers { permanent_uuid: "a204f0e9eec340e1bd4d74175dd392a2" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 33413 } }
I20260924 16:13:09.487222 6429 raft_consensus.cc:374] T c45fde87756a4a2ea51423eef9aa814a P a204f0e9eec340e1bd4d74175dd392a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:13:09.487349 6429 raft_consensus.cc:729] T c45fde87756a4a2ea51423eef9aa814a P a204f0e9eec340e1bd4d74175dd392a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a204f0e9eec340e1bd4d74175dd392a2, State: Initialized, Role: FOLLOWER
I20260924 16:13:09.487612 6429 consensus_queue.cc:260] T c45fde87756a4a2ea51423eef9aa814a P a204f0e9eec340e1bd4d74175dd392a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284ce6491d364d2a83dad33520490442" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 41013 } } peers { permanent_uuid: "43317d13dde547deabe618d1ea6110fa" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36571 } } peers { permanent_uuid: "a204f0e9eec340e1bd4d74175dd392a2" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 33413 } }
I20260924 16:13:09.488212 6429 ts_tablet_manager.cc:1428] T c45fde87756a4a2ea51423eef9aa814a P a204f0e9eec340e1bd4d74175dd392a2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260924 16:13:09.522466 6435 raft_consensus.cc:482] T c45fde87756a4a2ea51423eef9aa814a P 284ce6491d364d2a83dad33520490442 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:13:09.522706 6435 raft_consensus.cc:504] T c45fde87756a4a2ea51423eef9aa814a P 284ce6491d364d2a83dad33520490442 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284ce6491d364d2a83dad33520490442" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 41013 } } peers { permanent_uuid: "43317d13dde547deabe618d1ea6110fa" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36571 } } peers { permanent_uuid: "a204f0e9eec340e1bd4d74175dd392a2" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 33413 } }
I20260924 16:13:09.524519 6435 leader_election.cc:288] T c45fde87756a4a2ea51423eef9aa814a P 284ce6491d364d2a83dad33520490442 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43317d13dde547deabe618d1ea6110fa (127.29.234.2:36571), a204f0e9eec340e1bd4d74175dd392a2 (127.29.234.3:33413)
I20260924 16:13:09.532423 6512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:09.525174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.1:43633 (local address 127.29.234.2:36571)
0924 16:13:09.525873 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:09.525877 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:09.525892 (+ 15us) server_negotiation.cc:407] Connection header received
0924 16:13:09.525950 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:09.525954 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:09.526054 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:09.526186 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:13:09.526907 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:09.528001 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:09.530555 (+ 2554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:09.531484 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:09.531651 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:09.531728 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:09.531981 (+ 253us) server_negotiation.cc:299] Negotiation successful
0924 16:13:09.532112 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":419,"threads_started":1}
I20260924 16:13:09.532753 6510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:09.524761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.2:36571 (local address 127.29.234.1:43633)
0924 16:13:09.525410 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:09.525430 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:09.525630 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:09.526202 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:09.526207 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:09.526233 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:13:09.526697 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:09.526711 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:09.528194 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:09.528200 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:09.530364 (+ 2164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:09.530381 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:09.531635 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:09.531642 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:09.531818 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:09.531842 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 16:13:09.531955 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 16:13:09.532086 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":1711,"threads_started":1}
I20260924 16:13:09.533576 6424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:09.525213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.1:50165 (local address 127.29.234.3:33413)
0924 16:13:09.525406 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:09.525412 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:09.526934 (+ 1522us) server_negotiation.cc:407] Connection header received
0924 16:13:09.527121 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:09.527126 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:09.527213 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:09.527281 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:13:09.528061 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:09.528924 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:09.530812 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:09.532080 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:09.532278 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:09.532347 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:09.533149 (+ 802us) server_negotiation.cc:299] Negotiation successful
0924 16:13:09.533301 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 16:13:09.533479 6208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c45fde87756a4a2ea51423eef9aa814a" candidate_uuid: "284ce6491d364d2a83dad33520490442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43317d13dde547deabe618d1ea6110fa" is_pre_election: true
I20260924 16:13:09.533782 6208 raft_consensus.cc:2454] T c45fde87756a4a2ea51423eef9aa814a P 43317d13dde547deabe618d1ea6110fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 284ce6491d364d2a83dad33520490442 in term 0.
I20260924 16:13:09.534353 5999 leader_election.cc:302] T c45fde87756a4a2ea51423eef9aa814a P 284ce6491d364d2a83dad33520490442 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 284ce6491d364d2a83dad33520490442, 43317d13dde547deabe618d1ea6110fa; no voters:
I20260924 16:13:09.534533 6511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:09.524785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.3:33413 (local address 127.29.234.1:50165)
0924 16:13:09.526846 (+ 2061us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:09.526872 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:09.526971 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:09.527486 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:09.527490 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:09.527513 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:13:09.527879 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:09.527892 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:09.529131 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:09.529137 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:09.530651 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:09.530665 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:09.532522 (+ 1857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:09.532529 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:09.532749 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:09.532786 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 16:13:09.533098 (+ 312us) client_negotiation.cc:240] Negotiation successful
0924 16:13:09.534075 (+ 977us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1923,"spinlock_wait_cycles":30464,"thread_start_us":605,"threads_started":1}
I20260924 16:13:09.534777 6435 raft_consensus.cc:2781] T c45fde87756a4a2ea51423eef9aa814a P 284ce6491d364d2a83dad33520490442 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:13:09.534919 6435 raft_consensus.cc:482] T c45fde87756a4a2ea51423eef9aa814a P 284ce6491d364d2a83dad33520490442 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:13:09.535032 6435 raft_consensus.cc:3037] T c45fde87756a4a2ea51423eef9aa814a P 284ce6491d364d2a83dad33520490442 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:13:09.535146 6351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c45fde87756a4a2ea51423eef9aa814a" candidate_uuid: "284ce6491d364d2a83dad33520490442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a204f0e9eec340e1bd4d74175dd392a2" is_pre_election: true
I20260924 16:13:09.535423 6351 raft_consensus.cc:2454] T c45fde87756a4a2ea51423eef9aa814a P a204f0e9eec340e1bd4d74175dd392a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 284ce6491d364d2a83dad33520490442 in term 0.
I20260924 16:13:09.537339 6435 raft_consensus.cc:504] T c45fde87756a4a2ea51423eef9aa814a P 284ce6491d364d2a83dad33520490442 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284ce6491d364d2a83dad33520490442" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 41013 } } peers { permanent_uuid: "43317d13dde547deabe618d1ea6110fa" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36571 } } peers { permanent_uuid: "a204f0e9eec340e1bd4d74175dd392a2" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 33413 } }
I20260924 16:13:09.537993 6435 leader_election.cc:288] T c45fde87756a4a2ea51423eef9aa814a P 284ce6491d364d2a83dad33520490442 [CANDIDATE]: Term 1 election: Requested vote from peers 43317d13dde547deabe618d1ea6110fa (127.29.234.2:36571), a204f0e9eec340e1bd4d74175dd392a2 (127.29.234.3:33413)
I20260924 16:13:09.538475 6351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c45fde87756a4a2ea51423eef9aa814a" candidate_uuid: "284ce6491d364d2a83dad33520490442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a204f0e9eec340e1bd4d74175dd392a2"
I20260924 16:13:09.538451 6208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c45fde87756a4a2ea51423eef9aa814a" candidate_uuid: "284ce6491d364d2a83dad33520490442" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43317d13dde547deabe618d1ea6110fa"
I20260924 16:13:09.538676 6351 raft_consensus.cc:3037] T c45fde87756a4a2ea51423eef9aa814a P a204f0e9eec340e1bd4d74175dd392a2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:13:09.538687 6208 raft_consensus.cc:3037] T c45fde87756a4a2ea51423eef9aa814a P 43317d13dde547deabe618d1ea6110fa [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:13:09.541409 6351 raft_consensus.cc:2454] T c45fde87756a4a2ea51423eef9aa814a P a204f0e9eec340e1bd4d74175dd392a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 284ce6491d364d2a83dad33520490442 in term 1.
I20260924 16:13:09.541699 6208 raft_consensus.cc:2454] T c45fde87756a4a2ea51423eef9aa814a P 43317d13dde547deabe618d1ea6110fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 284ce6491d364d2a83dad33520490442 in term 1.
I20260924 16:13:09.541805 5997 leader_election.cc:302] T c45fde87756a4a2ea51423eef9aa814a P 284ce6491d364d2a83dad33520490442 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 284ce6491d364d2a83dad33520490442, a204f0e9eec340e1bd4d74175dd392a2; no voters:
I20260924 16:13:09.542129 6435 raft_consensus.cc:2781] T c45fde87756a4a2ea51423eef9aa814a P 284ce6491d364d2a83dad33520490442 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:13:09.542623 6435 raft_consensus.cc:686] T c45fde87756a4a2ea51423eef9aa814a P 284ce6491d364d2a83dad33520490442 [term 1 LEADER]: Becoming Leader. State: Replica: 284ce6491d364d2a83dad33520490442, State: Running, Role: LEADER
I20260924 16:13:09.543005 6435 consensus_queue.cc:237] T c45fde87756a4a2ea51423eef9aa814a P 284ce6491d364d2a83dad33520490442 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284ce6491d364d2a83dad33520490442" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 41013 } } peers { permanent_uuid: "43317d13dde547deabe618d1ea6110fa" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36571 } } peers { permanent_uuid: "a204f0e9eec340e1bd4d74175dd392a2" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 33413 } }
I20260924 16:13:09.546518 5907 catalog_manager.cc:5495] T c45fde87756a4a2ea51423eef9aa814a P 284ce6491d364d2a83dad33520490442 reported cstate change: term changed from 0 to 1, leader changed from <none> to 284ce6491d364d2a83dad33520490442 (127.29.234.1). New cstate: current_term: 1 leader_uuid: "284ce6491d364d2a83dad33520490442" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284ce6491d364d2a83dad33520490442" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 41013 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43317d13dde547deabe618d1ea6110fa" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a204f0e9eec340e1bd4d74175dd392a2" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 33413 } health_report { overall_health: UNKNOWN } } }
I20260924 16:13:09.569429 6516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:09.562569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49392 (local address 127.29.234.1:41013)
0924 16:13:09.563199 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:09.563205 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:09.563231 (+ 26us) server_negotiation.cc:407] Connection header received
0924 16:13:09.563368 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:09.563373 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:09.563488 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:09.563701 (+ 213us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:13:09.563985 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:09.565172 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:09.566512 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:09.567155 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:09.567297 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:09.567746 (+ 449us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:13:09.568624 (+ 878us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:13:09.568793 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:09.568903 (+ 110us) server_negotiation.cc:299] Negotiation successful
0924 16:13:09.568998 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":335,"threads_started":1}
I20260924 16:13:09.573308 6208 raft_consensus.cc:1261] T c45fde87756a4a2ea51423eef9aa814a P 43317d13dde547deabe618d1ea6110fa [term 1 FOLLOWER]: Refusing update from remote peer 284ce6491d364d2a83dad33520490442: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:13:09.573796 6351 raft_consensus.cc:1261] T c45fde87756a4a2ea51423eef9aa814a P a204f0e9eec340e1bd4d74175dd392a2 [term 1 FOLLOWER]: Refusing update from remote peer 284ce6491d364d2a83dad33520490442: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:13:09.574581 6513 consensus_queue.cc:1035] T c45fde87756a4a2ea51423eef9aa814a P 284ce6491d364d2a83dad33520490442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a204f0e9eec340e1bd4d74175dd392a2" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 33413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 16:13:09.576196 6435 consensus_queue.cc:1035] T c45fde87756a4a2ea51423eef9aa814a P 284ce6491d364d2a83dad33520490442 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43317d13dde547deabe618d1ea6110fa" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:13:09.610847 5908 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49472:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:13:09.611137 5908 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49472:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:13:09.614594 5908 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 441441046a974081ab6e0e4a1576c8f7: Sending DeleteTablet for 3 replicas of tablet c45fde87756a4a2ea51423eef9aa814a
I20260924 16:13:09.615427 6048 tablet_service.cc:1514] Processing DeleteTablet for tablet c45fde87756a4a2ea51423eef9aa814a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:13:09 UTC) from {username='oryx', principal='oryx/127.29.234.62@KRBTEST.COM'} at 127.0.0.1:49384
I20260924 16:13:09.615756 6188 tablet_service.cc:1514] Processing DeleteTablet for tablet c45fde87756a4a2ea51423eef9aa814a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:13:09 UTC) from {username='oryx', principal='oryx/127.29.234.62@KRBTEST.COM'} at 127.0.0.1:56866
I20260924 16:13:09.616077 6492 tablet_replica.cc:330] T c45fde87756a4a2ea51423eef9aa814a P 284ce6491d364d2a83dad33520490442: stopping tablet replica
I20260924 16:13:09.616302 6492 raft_consensus.cc:2229] T c45fde87756a4a2ea51423eef9aa814a P 284ce6491d364d2a83dad33520490442 [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:13:09.616776 6492 raft_consensus.cc:2258] T c45fde87756a4a2ea51423eef9aa814a P 284ce6491d364d2a83dad33520490442 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:13:09.617074 6493 tablet_replica.cc:330] T c45fde87756a4a2ea51423eef9aa814a P 43317d13dde547deabe618d1ea6110fa: stopping tablet replica
I20260924 16:13:09.617266 6493 raft_consensus.cc:2229] T c45fde87756a4a2ea51423eef9aa814a P 43317d13dde547deabe618d1ea6110fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:13:09.617506 6493 raft_consensus.cc:2258] T c45fde87756a4a2ea51423eef9aa814a P 43317d13dde547deabe618d1ea6110fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:13:09.618558 6492 ts_tablet_manager.cc:1905] T c45fde87756a4a2ea51423eef9aa814a P 284ce6491d364d2a83dad33520490442: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:13:09.624717 6493 ts_tablet_manager.cc:1905] T c45fde87756a4a2ea51423eef9aa814a P 43317d13dde547deabe618d1ea6110fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:13:09.625991 6492 ts_tablet_manager.cc:1918] T c45fde87756a4a2ea51423eef9aa814a P 284ce6491d364d2a83dad33520490442: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:13:09.626184 6492 log.cc:1196] T c45fde87756a4a2ea51423eef9aa814a P 284ce6491d364d2a83dad33520490442: Deleting WAL directory at /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/minicluster-data/ts-0/wal/wals/c45fde87756a4a2ea51423eef9aa814a
I20260924 16:13:09.626730 6492 ts_tablet_manager.cc:1939] T c45fde87756a4a2ea51423eef9aa814a P 284ce6491d364d2a83dad33520490442: Deleting consensus metadata
I20260924 16:13:09.629750 5892 catalog_manager.cc:4841] TS 284ce6491d364d2a83dad33520490442 (127.29.234.1:41013): tablet c45fde87756a4a2ea51423eef9aa814a (table test-table [id=f50c33298c4f4cb4b87f1eb863f6575c]) successfully deleted
I20260924 16:13:09.630765 6330 tablet_service.cc:1514] Processing DeleteTablet for tablet c45fde87756a4a2ea51423eef9aa814a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:13:09 UTC) from {username='oryx', principal='oryx/127.29.234.62@KRBTEST.COM'} at 127.0.0.1:50420
I20260924 16:13:09.631289 6494 tablet_replica.cc:330] T c45fde87756a4a2ea51423eef9aa814a P a204f0e9eec340e1bd4d74175dd392a2: stopping tablet replica
I20260924 16:13:09.631541 6494 raft_consensus.cc:2229] T c45fde87756a4a2ea51423eef9aa814a P a204f0e9eec340e1bd4d74175dd392a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:13:09.631698 6493 ts_tablet_manager.cc:1918] T c45fde87756a4a2ea51423eef9aa814a P 43317d13dde547deabe618d1ea6110fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:13:09.631821 6494 raft_consensus.cc:2258] T c45fde87756a4a2ea51423eef9aa814a P a204f0e9eec340e1bd4d74175dd392a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:13:09.631892 6493 log.cc:1196] T c45fde87756a4a2ea51423eef9aa814a P 43317d13dde547deabe618d1ea6110fa: Deleting WAL directory at /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/minicluster-data/ts-1/wal/wals/c45fde87756a4a2ea51423eef9aa814a
I20260924 16:13:09.632712 5908 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49472:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 16:13:09.633124 6494 ts_tablet_manager.cc:1905] T c45fde87756a4a2ea51423eef9aa814a P a204f0e9eec340e1bd4d74175dd392a2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 16:13:09.633456 5908 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:13:09.640196 6494 ts_tablet_manager.cc:1918] T c45fde87756a4a2ea51423eef9aa814a P a204f0e9eec340e1bd4d74175dd392a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:13:09.640846 6493 ts_tablet_manager.cc:1939] T c45fde87756a4a2ea51423eef9aa814a P 43317d13dde547deabe618d1ea6110fa: Deleting consensus metadata
I20260924 16:13:09.641075 6494 log.cc:1196] T c45fde87756a4a2ea51423eef9aa814a P a204f0e9eec340e1bd4d74175dd392a2: Deleting WAL directory at /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/minicluster-data/ts-2/wal/wals/c45fde87756a4a2ea51423eef9aa814a
I20260924 16:13:09.641858 6188 tablet_service.cc:1467] Processing CreateTablet for tablet 276fc38128e84d8aa832db0bd6a65a7b (DEFAULT_TABLE table=test-table [id=264270c51f0544f2af194e2798cadb39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:13:09.642306 6188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 276fc38128e84d8aa832db0bd6a65a7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:13:09.642446 6494 ts_tablet_manager.cc:1939] T c45fde87756a4a2ea51423eef9aa814a P a204f0e9eec340e1bd4d74175dd392a2: Deleting consensus metadata
I20260924 16:13:09.644315 6330 tablet_service.cc:1467] Processing CreateTablet for tablet 276fc38128e84d8aa832db0bd6a65a7b (DEFAULT_TABLE table=test-table [id=264270c51f0544f2af194e2798cadb39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:13:09.644783 6330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 276fc38128e84d8aa832db0bd6a65a7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:13:09.648944 5893 catalog_manager.cc:4841] TS 43317d13dde547deabe618d1ea6110fa (127.29.234.2:36571): tablet c45fde87756a4a2ea51423eef9aa814a (table test-table [id=f50c33298c4f4cb4b87f1eb863f6575c]) successfully deleted
I20260924 16:13:09.649401 6430 tablet_bootstrap.cc:492] T 276fc38128e84d8aa832db0bd6a65a7b P 43317d13dde547deabe618d1ea6110fa: Bootstrap starting.
I20260924 16:13:09.651096 6430 tablet_bootstrap.cc:654] T 276fc38128e84d8aa832db0bd6a65a7b P 43317d13dde547deabe618d1ea6110fa: Neither blocks nor log segments found. Creating new log.
I20260924 16:13:09.652123 5891 catalog_manager.cc:4841] TS a204f0e9eec340e1bd4d74175dd392a2 (127.29.234.3:33413): tablet c45fde87756a4a2ea51423eef9aa814a (table test-table [id=f50c33298c4f4cb4b87f1eb863f6575c]) successfully deleted
I20260924 16:13:09.652968 6048 tablet_service.cc:1467] Processing CreateTablet for tablet 276fc38128e84d8aa832db0bd6a65a7b (DEFAULT_TABLE table=test-table [id=264270c51f0544f2af194e2798cadb39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:13:09.653374 6429 tablet_bootstrap.cc:492] T 276fc38128e84d8aa832db0bd6a65a7b P a204f0e9eec340e1bd4d74175dd392a2: Bootstrap starting.
I20260924 16:13:09.653474 6048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 276fc38128e84d8aa832db0bd6a65a7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:13:09.655049 6429 tablet_bootstrap.cc:654] T 276fc38128e84d8aa832db0bd6a65a7b P a204f0e9eec340e1bd4d74175dd392a2: Neither blocks nor log segments found. Creating new log.
I20260924 16:13:09.661173 6428 tablet_bootstrap.cc:492] T 276fc38128e84d8aa832db0bd6a65a7b P 284ce6491d364d2a83dad33520490442: Bootstrap starting.
I20260924 16:13:09.663040 6428 tablet_bootstrap.cc:654] T 276fc38128e84d8aa832db0bd6a65a7b P 284ce6491d364d2a83dad33520490442: Neither blocks nor log segments found. Creating new log.
I20260924 16:13:09.670928 6428 tablet_bootstrap.cc:492] T 276fc38128e84d8aa832db0bd6a65a7b P 284ce6491d364d2a83dad33520490442: No bootstrap required, opened a new log
I20260924 16:13:09.671185 6428 ts_tablet_manager.cc:1397] T 276fc38128e84d8aa832db0bd6a65a7b P 284ce6491d364d2a83dad33520490442: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.002s
I20260924 16:13:09.671214 6429 tablet_bootstrap.cc:492] T 276fc38128e84d8aa832db0bd6a65a7b P a204f0e9eec340e1bd4d74175dd392a2: No bootstrap required, opened a new log
I20260924 16:13:09.671447 6429 ts_tablet_manager.cc:1397] T 276fc38128e84d8aa832db0bd6a65a7b P a204f0e9eec340e1bd4d74175dd392a2: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.010s
I20260924 16:13:09.672158 6428 raft_consensus.cc:348] T 276fc38128e84d8aa832db0bd6a65a7b P 284ce6491d364d2a83dad33520490442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284ce6491d364d2a83dad33520490442" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 41013 } } peers { permanent_uuid: "43317d13dde547deabe618d1ea6110fa" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36571 } } peers { permanent_uuid: "a204f0e9eec340e1bd4d74175dd392a2" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 33413 } }
I20260924 16:13:09.672524 6428 raft_consensus.cc:374] T 276fc38128e84d8aa832db0bd6a65a7b P 284ce6491d364d2a83dad33520490442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:13:09.672644 6428 raft_consensus.cc:729] T 276fc38128e84d8aa832db0bd6a65a7b P 284ce6491d364d2a83dad33520490442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 284ce6491d364d2a83dad33520490442, State: Initialized, Role: FOLLOWER
I20260924 16:13:09.673018 6428 consensus_queue.cc:260] T 276fc38128e84d8aa832db0bd6a65a7b P 284ce6491d364d2a83dad33520490442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284ce6491d364d2a83dad33520490442" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 41013 } } peers { permanent_uuid: "43317d13dde547deabe618d1ea6110fa" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36571 } } peers { permanent_uuid: "a204f0e9eec340e1bd4d74175dd392a2" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 33413 } }
I20260924 16:13:09.673657 6428 ts_tablet_manager.cc:1428] T 276fc38128e84d8aa832db0bd6a65a7b P 284ce6491d364d2a83dad33520490442: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 16:13:09.673702 6429 raft_consensus.cc:348] T 276fc38128e84d8aa832db0bd6a65a7b P a204f0e9eec340e1bd4d74175dd392a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284ce6491d364d2a83dad33520490442" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 41013 } } peers { permanent_uuid: "43317d13dde547deabe618d1ea6110fa" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36571 } } peers { permanent_uuid: "a204f0e9eec340e1bd4d74175dd392a2" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 33413 } }
I20260924 16:13:09.675289 6430 tablet_bootstrap.cc:492] T 276fc38128e84d8aa832db0bd6a65a7b P 43317d13dde547deabe618d1ea6110fa: No bootstrap required, opened a new log
I20260924 16:13:09.675508 6430 ts_tablet_manager.cc:1397] T 276fc38128e84d8aa832db0bd6a65a7b P 43317d13dde547deabe618d1ea6110fa: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.006s
I20260924 16:13:09.676383 6430 raft_consensus.cc:348] T 276fc38128e84d8aa832db0bd6a65a7b P 43317d13dde547deabe618d1ea6110fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284ce6491d364d2a83dad33520490442" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 41013 } } peers { permanent_uuid: "43317d13dde547deabe618d1ea6110fa" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36571 } } peers { permanent_uuid: "a204f0e9eec340e1bd4d74175dd392a2" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 33413 } }
I20260924 16:13:09.676745 6430 raft_consensus.cc:374] T 276fc38128e84d8aa832db0bd6a65a7b P 43317d13dde547deabe618d1ea6110fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:13:09.676874 6430 raft_consensus.cc:729] T 276fc38128e84d8aa832db0bd6a65a7b P 43317d13dde547deabe618d1ea6110fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43317d13dde547deabe618d1ea6110fa, State: Initialized, Role: FOLLOWER
I20260924 16:13:09.677197 6430 consensus_queue.cc:260] T 276fc38128e84d8aa832db0bd6a65a7b P 43317d13dde547deabe618d1ea6110fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284ce6491d364d2a83dad33520490442" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 41013 } } peers { permanent_uuid: "43317d13dde547deabe618d1ea6110fa" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36571 } } peers { permanent_uuid: "a204f0e9eec340e1bd4d74175dd392a2" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 33413 } }
I20260924 16:13:09.678769 6430 ts_tablet_manager.cc:1428] T 276fc38128e84d8aa832db0bd6a65a7b P 43317d13dde547deabe618d1ea6110fa: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 16:13:09.679044 6429 raft_consensus.cc:374] T 276fc38128e84d8aa832db0bd6a65a7b P a204f0e9eec340e1bd4d74175dd392a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:13:09.679154 6429 raft_consensus.cc:729] T 276fc38128e84d8aa832db0bd6a65a7b P a204f0e9eec340e1bd4d74175dd392a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a204f0e9eec340e1bd4d74175dd392a2, State: Initialized, Role: FOLLOWER
I20260924 16:13:09.679433 6429 consensus_queue.cc:260] T 276fc38128e84d8aa832db0bd6a65a7b P a204f0e9eec340e1bd4d74175dd392a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284ce6491d364d2a83dad33520490442" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 41013 } } peers { permanent_uuid: "43317d13dde547deabe618d1ea6110fa" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36571 } } peers { permanent_uuid: "a204f0e9eec340e1bd4d74175dd392a2" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 33413 } }
I20260924 16:13:09.683400 6429 ts_tablet_manager.cc:1428] T 276fc38128e84d8aa832db0bd6a65a7b P a204f0e9eec340e1bd4d74175dd392a2: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20260924 16:13:09.733263 6437 raft_consensus.cc:482] T 276fc38128e84d8aa832db0bd6a65a7b P 43317d13dde547deabe618d1ea6110fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:13:09.733525 6437 raft_consensus.cc:504] T 276fc38128e84d8aa832db0bd6a65a7b P 43317d13dde547deabe618d1ea6110fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284ce6491d364d2a83dad33520490442" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 41013 } } peers { permanent_uuid: "43317d13dde547deabe618d1ea6110fa" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36571 } } peers { permanent_uuid: "a204f0e9eec340e1bd4d74175dd392a2" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 33413 } }
I20260924 16:13:09.734170 6437 leader_election.cc:288] T 276fc38128e84d8aa832db0bd6a65a7b P 43317d13dde547deabe618d1ea6110fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 284ce6491d364d2a83dad33520490442 (127.29.234.1:41013), a204f0e9eec340e1bd4d74175dd392a2 (127.29.234.3:33413)
I20260924 16:13:09.734815 6072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "276fc38128e84d8aa832db0bd6a65a7b" candidate_uuid: "43317d13dde547deabe618d1ea6110fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "284ce6491d364d2a83dad33520490442" is_pre_election: true
I20260924 16:13:09.734870 6351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "276fc38128e84d8aa832db0bd6a65a7b" candidate_uuid: "43317d13dde547deabe618d1ea6110fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a204f0e9eec340e1bd4d74175dd392a2" is_pre_election: true
I20260924 16:13:09.735076 6072 raft_consensus.cc:2454] T 276fc38128e84d8aa832db0bd6a65a7b P 284ce6491d364d2a83dad33520490442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43317d13dde547deabe618d1ea6110fa in term 0.
I20260924 16:13:09.735150 6351 raft_consensus.cc:2454] T 276fc38128e84d8aa832db0bd6a65a7b P a204f0e9eec340e1bd4d74175dd392a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43317d13dde547deabe618d1ea6110fa in term 0.
I20260924 16:13:09.735579 6139 leader_election.cc:302] T 276fc38128e84d8aa832db0bd6a65a7b P 43317d13dde547deabe618d1ea6110fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43317d13dde547deabe618d1ea6110fa, a204f0e9eec340e1bd4d74175dd392a2; no voters:
I20260924 16:13:09.735908 6437 raft_consensus.cc:2781] T 276fc38128e84d8aa832db0bd6a65a7b P 43317d13dde547deabe618d1ea6110fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:13:09.736018 6437 raft_consensus.cc:482] T 276fc38128e84d8aa832db0bd6a65a7b P 43317d13dde547deabe618d1ea6110fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:13:09.736124 6437 raft_consensus.cc:3037] T 276fc38128e84d8aa832db0bd6a65a7b P 43317d13dde547deabe618d1ea6110fa [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:13:09.739180 6437 raft_consensus.cc:504] T 276fc38128e84d8aa832db0bd6a65a7b P 43317d13dde547deabe618d1ea6110fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284ce6491d364d2a83dad33520490442" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 41013 } } peers { permanent_uuid: "43317d13dde547deabe618d1ea6110fa" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36571 } } peers { permanent_uuid: "a204f0e9eec340e1bd4d74175dd392a2" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 33413 } }
I20260924 16:13:09.739888 6437 leader_election.cc:288] T 276fc38128e84d8aa832db0bd6a65a7b P 43317d13dde547deabe618d1ea6110fa [CANDIDATE]: Term 1 election: Requested vote from peers 284ce6491d364d2a83dad33520490442 (127.29.234.1:41013), a204f0e9eec340e1bd4d74175dd392a2 (127.29.234.3:33413)
I20260924 16:13:09.740386 6072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "276fc38128e84d8aa832db0bd6a65a7b" candidate_uuid: "43317d13dde547deabe618d1ea6110fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "284ce6491d364d2a83dad33520490442"
I20260924 16:13:09.740391 6351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "276fc38128e84d8aa832db0bd6a65a7b" candidate_uuid: "43317d13dde547deabe618d1ea6110fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a204f0e9eec340e1bd4d74175dd392a2"
I20260924 16:13:09.740597 6351 raft_consensus.cc:3037] T 276fc38128e84d8aa832db0bd6a65a7b P a204f0e9eec340e1bd4d74175dd392a2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:13:09.740603 6072 raft_consensus.cc:3037] T 276fc38128e84d8aa832db0bd6a65a7b P 284ce6491d364d2a83dad33520490442 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:13:09.742836 6072 raft_consensus.cc:2454] T 276fc38128e84d8aa832db0bd6a65a7b P 284ce6491d364d2a83dad33520490442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43317d13dde547deabe618d1ea6110fa in term 1.
I20260924 16:13:09.743330 6140 leader_election.cc:302] T 276fc38128e84d8aa832db0bd6a65a7b P 43317d13dde547deabe618d1ea6110fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 284ce6491d364d2a83dad33520490442, 43317d13dde547deabe618d1ea6110fa; no voters:
I20260924 16:13:09.743594 6351 raft_consensus.cc:2454] T 276fc38128e84d8aa832db0bd6a65a7b P a204f0e9eec340e1bd4d74175dd392a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43317d13dde547deabe618d1ea6110fa in term 1.
I20260924 16:13:09.743631 6437 raft_consensus.cc:2781] T 276fc38128e84d8aa832db0bd6a65a7b P 43317d13dde547deabe618d1ea6110fa [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:13:09.751858 6437 raft_consensus.cc:686] T 276fc38128e84d8aa832db0bd6a65a7b P 43317d13dde547deabe618d1ea6110fa [term 1 LEADER]: Becoming Leader. State: Replica: 43317d13dde547deabe618d1ea6110fa, State: Running, Role: LEADER
I20260924 16:13:09.752375 6437 consensus_queue.cc:237] T 276fc38128e84d8aa832db0bd6a65a7b P 43317d13dde547deabe618d1ea6110fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284ce6491d364d2a83dad33520490442" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 41013 } } peers { permanent_uuid: "43317d13dde547deabe618d1ea6110fa" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36571 } } peers { permanent_uuid: "a204f0e9eec340e1bd4d74175dd392a2" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 33413 } }
I20260924 16:13:09.756278 5906 catalog_manager.cc:5495] T 276fc38128e84d8aa832db0bd6a65a7b P 43317d13dde547deabe618d1ea6110fa reported cstate change: term changed from 0 to 1, leader changed from <none> to 43317d13dde547deabe618d1ea6110fa (127.29.234.2). New cstate: current_term: 1 leader_uuid: "43317d13dde547deabe618d1ea6110fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284ce6491d364d2a83dad33520490442" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 41013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43317d13dde547deabe618d1ea6110fa" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a204f0e9eec340e1bd4d74175dd392a2" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 33413 } health_report { overall_health: UNKNOWN } } }
I20260924 16:13:09.782516 6512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:09.776930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56880 (local address 127.29.234.2:36571)
0924 16:13:09.777130 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:09.777134 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:09.777208 (+ 74us) server_negotiation.cc:407] Connection header received
0924 16:13:09.777488 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:09.777492 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:09.777569 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:09.777730 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:13:09.778114 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:09.778831 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:09.780141 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:09.780550 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:09.780687 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:09.781267 (+ 580us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:13:09.781942 (+ 675us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:13:09.782110 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:09.782167 (+ 57us) server_negotiation.cc:299] Negotiation successful
0924 16:13:09.782253 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 16:13:09.799428 6424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:09.792989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.2:38257 (local address 127.29.234.3:33413)
0924 16:13:09.793193 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:09.793200 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:09.793831 (+ 631us) server_negotiation.cc:407] Connection header received
0924 16:13:09.793986 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:09.793992 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:09.794081 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:09.794184 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:13:09.794619 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:09.795688 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:09.797174 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:09.798002 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:09.798234 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:09.798695 (+ 461us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:13:09.798832 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:13:09.798925 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:09.799050 (+ 125us) server_negotiation.cc:299] Negotiation successful
0924 16:13:09.799136 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 16:13:09.799517 6528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:09.792986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.3:33413 (local address 127.29.234.2:38257)
0924 16:13:09.793734 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:09.793750 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:09.793863 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:09.794200 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:09.794203 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:09.794214 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 16:13:09.794473 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:09.794480 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:09.795825 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:09.795829 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:09.796987 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:09.796998 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:09.798277 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:09.798282 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:09.798441 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:09.798503 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 16:13:09.798925 (+ 422us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 16:13:09.798934 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 16:13:09.799000 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 16:13:09.799178 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":420,"threads_started":1}
I20260924 16:13:09.809295 6351 raft_consensus.cc:1261] T 276fc38128e84d8aa832db0bd6a65a7b P a204f0e9eec340e1bd4d74175dd392a2 [term 1 FOLLOWER]: Refusing update from remote peer 43317d13dde547deabe618d1ea6110fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:13:09.809283 6072 raft_consensus.cc:1261] T 276fc38128e84d8aa832db0bd6a65a7b P 284ce6491d364d2a83dad33520490442 [term 1 FOLLOWER]: Refusing update from remote peer 43317d13dde547deabe618d1ea6110fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:13:09.809999 6524 consensus_queue.cc:1035] T 276fc38128e84d8aa832db0bd6a65a7b P 43317d13dde547deabe618d1ea6110fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "284ce6491d364d2a83dad33520490442" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 41013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:13:09.810417 6437 consensus_queue.cc:1035] T 276fc38128e84d8aa832db0bd6a65a7b P 43317d13dde547deabe618d1ea6110fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "a204f0e9eec340e1bd4d74175dd392a2" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 33413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:13:09.869779 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:09.862411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.2:36571 (local address 127.29.234.3:35913)
0924 16:13:09.863098 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:09.863114 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:09.863316 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:09.863887 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:09.863896 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:09.863916 (+ 20us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 16:13:09.864554 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:09.864565 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:09.866300 (+ 1735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:09.866304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:09.867602 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:09.867618 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:09.868623 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:09.868630 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:09.868785 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:09.868842 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 16:13:09.869275 (+ 433us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 16:13:09.869284 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 16:13:09.869400 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 16:13:09.869510 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":393,"threads_started":1}
I20260924 16:13:09.869966 6512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:09.862451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.3:35913 (local address 127.29.234.2:36571)
0924 16:13:09.862647 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:09.862657 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:09.863247 (+ 590us) server_negotiation.cc:407] Connection header received
0924 16:13:09.863458 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:09.863463 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:09.863544 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:09.863636 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:13:09.864728 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:09.866086 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:09.867754 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:09.868460 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:09.868609 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:09.869019 (+ 410us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:13:09.869173 (+ 154us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:13:09.869270 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:09.869448 (+ 178us) server_negotiation.cc:299] Negotiation successful
0924 16:13:09.869591 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 16:13:09.921195 5906 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49472:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:13:09.921553 5906 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49472:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:13:09.925637 5906 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 441441046a974081ab6e0e4a1576c8f7: Sending DeleteTablet for 3 replicas of tablet 276fc38128e84d8aa832db0bd6a65a7b
I20260924 16:13:09.926843 6188 tablet_service.cc:1514] Processing DeleteTablet for tablet 276fc38128e84d8aa832db0bd6a65a7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:13:09 UTC) from {username='oryx', principal='oryx/127.29.234.62@KRBTEST.COM'} at 127.0.0.1:56866
I20260924 16:13:09.926884 6048 tablet_service.cc:1514] Processing DeleteTablet for tablet 276fc38128e84d8aa832db0bd6a65a7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:13:09 UTC) from {username='oryx', principal='oryx/127.29.234.62@KRBTEST.COM'} at 127.0.0.1:49384
I20260924 16:13:09.927141 6493 tablet_replica.cc:330] T 276fc38128e84d8aa832db0bd6a65a7b P 43317d13dde547deabe618d1ea6110fa: stopping tablet replica
I20260924 16:13:09.927235 6492 tablet_replica.cc:330] T 276fc38128e84d8aa832db0bd6a65a7b P 284ce6491d364d2a83dad33520490442: stopping tablet replica
I20260924 16:13:09.927340 6493 raft_consensus.cc:2229] T 276fc38128e84d8aa832db0bd6a65a7b P 43317d13dde547deabe618d1ea6110fa [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:13:09.927443 6492 raft_consensus.cc:2229] T 276fc38128e84d8aa832db0bd6a65a7b P 284ce6491d364d2a83dad33520490442 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:13:09.927686 6492 raft_consensus.cc:2258] T 276fc38128e84d8aa832db0bd6a65a7b P 284ce6491d364d2a83dad33520490442 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:13:09.927891 6493 raft_consensus.cc:2258] T 276fc38128e84d8aa832db0bd6a65a7b P 43317d13dde547deabe618d1ea6110fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:13:09.928369 6330 tablet_service.cc:1514] Processing DeleteTablet for tablet 276fc38128e84d8aa832db0bd6a65a7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:13:09 UTC) from {username='oryx', principal='oryx/127.29.234.62@KRBTEST.COM'} at 127.0.0.1:50420
I20260924 16:13:09.928704 6494 tablet_replica.cc:330] T 276fc38128e84d8aa832db0bd6a65a7b P a204f0e9eec340e1bd4d74175dd392a2: stopping tablet replica
I20260924 16:13:09.928912 6494 raft_consensus.cc:2229] T 276fc38128e84d8aa832db0bd6a65a7b P a204f0e9eec340e1bd4d74175dd392a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:13:09.929131 6494 raft_consensus.cc:2258] T 276fc38128e84d8aa832db0bd6a65a7b P a204f0e9eec340e1bd4d74175dd392a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:13:09.929381 6493 ts_tablet_manager.cc:1905] T 276fc38128e84d8aa832db0bd6a65a7b P 43317d13dde547deabe618d1ea6110fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:13:09.930940 6494 ts_tablet_manager.cc:1905] T 276fc38128e84d8aa832db0bd6a65a7b P a204f0e9eec340e1bd4d74175dd392a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:13:09.936573 6493 ts_tablet_manager.cc:1918] T 276fc38128e84d8aa832db0bd6a65a7b P 43317d13dde547deabe618d1ea6110fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 16:13:09.936720 6493 log.cc:1196] T 276fc38128e84d8aa832db0bd6a65a7b P 43317d13dde547deabe618d1ea6110fa: Deleting WAL directory at /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/minicluster-data/ts-1/wal/wals/276fc38128e84d8aa832db0bd6a65a7b
I20260924 16:13:09.937180 6493 ts_tablet_manager.cc:1939] T 276fc38128e84d8aa832db0bd6a65a7b P 43317d13dde547deabe618d1ea6110fa: Deleting consensus metadata
I20260924 16:13:09.938163 6492 ts_tablet_manager.cc:1905] T 276fc38128e84d8aa832db0bd6a65a7b P 284ce6491d364d2a83dad33520490442: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:13:09.941727 5893 catalog_manager.cc:4841] TS 43317d13dde547deabe618d1ea6110fa (127.29.234.2:36571): tablet 276fc38128e84d8aa832db0bd6a65a7b (table test-table [id=264270c51f0544f2af194e2798cadb39]) successfully deleted
I20260924 16:13:09.943262 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 5981
I20260924 16:13:09.946688 6494 ts_tablet_manager.cc:1918] T 276fc38128e84d8aa832db0bd6a65a7b P a204f0e9eec340e1bd4d74175dd392a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 16:13:09.946939 6494 log.cc:1196] T 276fc38128e84d8aa832db0bd6a65a7b P a204f0e9eec340e1bd4d74175dd392a2: Deleting WAL directory at /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/minicluster-data/ts-2/wal/wals/276fc38128e84d8aa832db0bd6a65a7b
I20260924 16:13:09.947444 6494 ts_tablet_manager.cc:1939] T 276fc38128e84d8aa832db0bd6a65a7b P a204f0e9eec340e1bd4d74175dd392a2: Deleting consensus metadata
I20260924 16:13:09.948634 5891 catalog_manager.cc:4841] TS a204f0e9eec340e1bd4d74175dd392a2 (127.29.234.3:33413): tablet 276fc38128e84d8aa832db0bd6a65a7b (table test-table [id=264270c51f0544f2af194e2798cadb39]) successfully deleted
I20260924 16:13:09.951150 6492 ts_tablet_manager.cc:1918] T 276fc38128e84d8aa832db0bd6a65a7b P 284ce6491d364d2a83dad33520490442: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 16:13:09.951320 6492 log.cc:1196] T 276fc38128e84d8aa832db0bd6a65a7b P 284ce6491d364d2a83dad33520490442: Deleting WAL directory at /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790266323169505-30632-0/minicluster-data/ts-0/wal/wals/276fc38128e84d8aa832db0bd6a65a7b
I20260924 16:13:09.951860 6492 ts_tablet_manager.cc:1939] T 276fc38128e84d8aa832db0bd6a65a7b P 284ce6491d364d2a83dad33520490442: Deleting consensus metadata
I20260924 16:13:09.953074 5892 catalog_manager.cc:4841] TS 284ce6491d364d2a83dad33520490442 (127.29.234.1:41013): tablet 276fc38128e84d8aa832db0bd6a65a7b (table test-table [id=264270c51f0544f2af194e2798cadb39]) successfully deleted
Sep 24 16:13:09 dist-test-slave-v5j9 krb5kdc[5862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.29.234.1@KRBTEST.COM
I20260924 16:13:09.977478 6516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:09.944738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49396 (local address 127.29.234.1:41013)
0924 16:13:09.944970 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:09.944976 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:09.957644 (+ 12668us) server_negotiation.cc:407] Connection header received
0924 16:13:09.957852 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:09.957861 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:09.957963 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:09.958087 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:09.959635 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:09.960971 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:09.962906 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:09.963706 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:09.963858 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:09.969674 (+ 5816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:13:09.969689 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:13:09.969709 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:13:09.969754 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:13:09.974089 (+ 4335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:09.974992 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:09.975002 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:09.975006 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:09.975189 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:09.975616 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:09.975621 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:09.975624 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:09.976129 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:13:09.976444 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:09.976774 (+ 330us) server_negotiation.cc:299] Negotiation successful
0924 16:13:09.977004 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 16:13:09.978222 6114 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:13:10.120558 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 5981
I20260924 16:13:10.143051 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 6123
Sep 24 16:13:10 dist-test-slave-v5j9 krb5kdc[5862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.29.234.2@KRBTEST.COM
I20260924 16:13:10.159889 6512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:10.143907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56892 (local address 127.29.234.2:36571)
0924 16:13:10.144111 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:10.144117 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:10.144137 (+ 20us) server_negotiation.cc:407] Connection header received
0924 16:13:10.144284 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:10.144290 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:10.144370 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:10.144488 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:10.145705 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:10.146893 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:10.148064 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:10.148756 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:10.148879 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:10.153080 (+ 4201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:13:10.153096 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:13:10.153118 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:13:10.153167 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:13:10.156991 (+ 3824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:10.157745 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:10.157756 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:10.157760 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:10.157904 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:10.158238 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:10.158242 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:10.158246 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:10.158780 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:13:10.159050 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:10.159328 (+ 278us) server_negotiation.cc:299] Negotiation successful
0924 16:13:10.159545 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 16:13:10.160547 6254 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:13:10.282203 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 6123
I20260924 16:13:10.302721 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 6264
Sep 24 16:13:10 dist-test-slave-v5j9 krb5kdc[5862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.29.234.3@KRBTEST.COM
I20260924 16:13:10.318898 6544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:10.303530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50436 (local address 127.29.234.3:33413)
0924 16:13:10.304003 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:10.304007 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:10.304020 (+ 13us) server_negotiation.cc:407] Connection header received
0924 16:13:10.304073 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:10.304076 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:10.304153 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:10.304306 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:10.305704 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:10.306434 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:10.307714 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:10.308104 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:10.308233 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:10.312996 (+ 4763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:13:10.313011 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:13:10.313030 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:13:10.313063 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:13:10.315982 (+ 2919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:10.316638 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:10.316655 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:10.316658 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:10.316789 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:10.317169 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:10.317173 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:10.317177 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:10.317697 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:13:10.318026 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:10.318322 (+ 296us) server_negotiation.cc:299] Negotiation successful
0924 16:13:10.318547 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":230,"threads_started":1}
I20260924 16:13:10.319542 6398 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:13:10.474382 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 6264
I20260924 16:13:10.499833 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 5875
I20260924 16:13:10.501050 5950 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:13:10.637239 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 5875
2026-09-24T16:13:10Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (7050 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790266323169505-30632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:13:10 dist-test-slave-v5j9 krb5kdc[6550](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:13:10 dist-test-slave-v5j9 krb5kdc[6550](info): set up 2 sockets
Sep 24 16:13:10 dist-test-slave-v5j9 krb5kdc[6550](info): commencing operation
krb5kdc: starting...
W20260924 16:13:11.761883 30632 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Sep 24 16:13:11 dist-test-slave-v5j9 krb5kdc[6550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266391, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:13:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:13:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.234.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab.
Entry for principal kudu/127.29.234.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab.
I20260924 16:13:11.997316 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790266323169505-30632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790266323169505-30632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790266323169505-30632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:36781
--webserver_interface=127.29.234.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab
--principal=kudu/127.29.234.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:33261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.234.62:36781
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:13:12.329726 6563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:13:12.330017 6563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:13:12.330180 6563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:13:12.337462 6563 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:13:12.337568 6563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:13:12.337669 6563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:13:12.337769 6563 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:13:12.337862 6563 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:13:12.347937 6563 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:33261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790266323169505-30632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790266323169505-30632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.234.62:36781
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:36781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.234.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790266323169505-30632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:13:12.349251 6563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:13:12.350633 6563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:13:12.363066 6568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:13:12.363010 6569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:13:12.363128 6563 server_base.cc:1034] running on GCE node
W20260924 16:13:12.363189 6572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:13:12.364373 6563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:13:12.365767 6563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:13:12.367008 6563 hybrid_clock.cc:648] HybridClock initialized: now 1790266392366954 us; error 44 us; skew 500 ppm
I20260924 16:13:12.367357 6563 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-24T16:13:12Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1807 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:13:12 dist-test-slave-v5j9 krb5kdc[6578](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:13:12 dist-test-slave-v5j9 krb5kdc[6578](info): set up 2 sockets
Sep 24 16:13:12 dist-test-slave-v5j9 krb5kdc[6578](info): commencing operation
krb5kdc: starting...
W20260924 16:13:13.549850 30632 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.003s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Sep 24 16:13:13 dist-test-slave-v5j9 krb5kdc[6578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:13:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:13:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.234.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab.
Entry for principal kudu/127.29.234.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab.
I20260924 16:13:13.779307 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:37997
--webserver_interface=127.29.234.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab
--principal=kudu/127.29.234.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:44393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.234.62:37997
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:13:14.110679 6591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:13:14.111011 6591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:13:14.111202 6591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:13:14.119218 6591 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:13:14.119349 6591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:13:14.119472 6591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:13:14.119556 6591 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:13:14.119642 6591 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:13:14.130004 6591 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:44393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.234.62:37997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab
--principal=kudu/127.29.234.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:37997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.234.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:13:14.131490 6591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:13:14.132908 6591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:13:14.145372 6596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:13:14.145327 6600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:13:14.145159 6597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:13:14.145766 6591 server_base.cc:1034] running on GCE node
I20260924 16:13:14.146772 6591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:13:14.148190 6591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:13:14.149456 6591 hybrid_clock.cc:648] HybridClock initialized: now 1790266394149395 us; error 41 us; skew 500 ppm
I20260924 16:13:14.149829 6591 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:13:14 dist-test-slave-v5j9 krb5kdc[6578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266394, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:13:14.161283 6591 init.cc:377] Logged in from keytab as kudu/127.29.234.62@KRBTEST.COM (short username kudu)
I20260924 16:13:14.163125 6591 webserver.cc:460] Webserver started at http://127.29.234.62:41429/ using document root <none> and password file <none>
I20260924 16:13:14.163887 6591 fs_manager.cc:359] Metadata directory not provided
I20260924 16:13:14.164036 6591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:13:14.164402 6591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:13:14.167707 6591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/master-0/data/instance:
uuid: "c4607b3be6854d73853b8976e4797024"
format_stamp: "Formatted at 2026-09-24 16:13:14 on dist-test-slave-v5j9"
server_key: "efb0f67c24860fb8e4b78709e1adabc2"
server_key_iv: "afb603754911af58e8acc24dd6b36439"
server_key_version: "encryptionkey@0"
I20260924 16:13:14.168537 6591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/master-0/wal/instance:
uuid: "c4607b3be6854d73853b8976e4797024"
format_stamp: "Formatted at 2026-09-24 16:13:14 on dist-test-slave-v5j9"
server_key: "efb0f67c24860fb8e4b78709e1adabc2"
server_key_iv: "afb603754911af58e8acc24dd6b36439"
server_key_version: "encryptionkey@0"
I20260924 16:13:14.173827 6591 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260924 16:13:14.177327 6607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:13:14.178756 6591 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:13:14.178946 6591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/master-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/master-0/wal
uuid: "c4607b3be6854d73853b8976e4797024"
format_stamp: "Formatted at 2026-09-24 16:13:14 on dist-test-slave-v5j9"
server_key: "efb0f67c24860fb8e4b78709e1adabc2"
server_key_iv: "afb603754911af58e8acc24dd6b36439"
server_key_version: "encryptionkey@0"
I20260924 16:13:14.179159 6591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:13:14.201363 6591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:13:14.205566 6591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:13:14.205869 6591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:13:14.225562 6591 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.62:37997
I20260924 16:13:14.225662 6658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.62:37997 every 8 connection(s)
I20260924 16:13:14.227020 6591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
I20260924 16:13:14.230881 6659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:13:14.231909 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 6591
I20260924 16:13:14.232143 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/master-0/wal/instance
I20260924 16:13:14.233001 30632 external_mini_cluster.cc:1428] Setting key c59adc560eac2592ce9dad23cb8781e8
I20260924 16:13:14.245501 6659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4607b3be6854d73853b8976e4797024: Bootstrap starting.
Sep 24 16:13:14 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:13:14.249109 6659 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4607b3be6854d73853b8976e4797024: Neither blocks nor log segments found. Creating new log.
I20260924 16:13:14.250298 6659 log.cc:824] T 00000000000000000000000000000000 P c4607b3be6854d73853b8976e4797024: Log is configured to *not* fsync() on all Append() calls
I20260924 16:13:14.253198 6659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4607b3be6854d73853b8976e4797024: No bootstrap required, opened a new log
I20260924 16:13:14.255882 6662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:14.235119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38192 (local address 127.29.234.62:37997)
0924 16:13:14.235967 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:14.235978 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:14.236021 (+ 43us) server_negotiation.cc:407] Connection header received
0924 16:13:14.237717 (+ 1696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:14.237748 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:14.238140 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:14.238479 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:14.240294 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:14.241635 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:14.243313 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:14.243936 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:14.244055 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:14.248552 (+ 4497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:13:14.248561 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:13:14.248578 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:13:14.248615 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:13:14.251786 (+ 3171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:14.252491 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:14.252497 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:14.252503 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:14.252667 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:14.253058 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:14.253063 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:14.253066 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:14.253659 (+ 593us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:13:14.254002 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:14.254441 (+ 439us) server_negotiation.cc:299] Negotiation successful
0924 16:13:14.254871 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":372,"threads_started":1}
I20260924 16:13:14.261169 6659 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4607b3be6854d73853b8976e4797024 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4607b3be6854d73853b8976e4797024" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 37997 } }
I20260924 16:13:14.261612 6659 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4607b3be6854d73853b8976e4797024 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:13:14.261721 6659 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4607b3be6854d73853b8976e4797024 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4607b3be6854d73853b8976e4797024, State: Initialized, Role: FOLLOWER
I20260924 16:13:14.262326 6659 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4607b3be6854d73853b8976e4797024 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4607b3be6854d73853b8976e4797024" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 37997 } }
I20260924 16:13:14.262583 6659 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4607b3be6854d73853b8976e4797024 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:13:14.262734 6659 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4607b3be6854d73853b8976e4797024 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:13:14.262912 6659 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4607b3be6854d73853b8976e4797024 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:13:14.265473 6659 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4607b3be6854d73853b8976e4797024 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4607b3be6854d73853b8976e4797024" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 37997 } }
I20260924 16:13:14.265995 6659 leader_election.cc:302] T 00000000000000000000000000000000 P c4607b3be6854d73853b8976e4797024 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4607b3be6854d73853b8976e4797024; no voters:
I20260924 16:13:14.266624 6659 leader_election.cc:288] T 00000000000000000000000000000000 P c4607b3be6854d73853b8976e4797024 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:13:14.266759 6664 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4607b3be6854d73853b8976e4797024 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:13:14.267449 6664 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4607b3be6854d73853b8976e4797024 [term 1 LEADER]: Becoming Leader. State: Replica: c4607b3be6854d73853b8976e4797024, State: Running, Role: LEADER
I20260924 16:13:14.267970 6664 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4607b3be6854d73853b8976e4797024 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4607b3be6854d73853b8976e4797024" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 37997 } }
I20260924 16:13:14.268739 6659 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4607b3be6854d73853b8976e4797024 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:13:14.270782 6666 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4607b3be6854d73853b8976e4797024 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4607b3be6854d73853b8976e4797024. Latest consensus state: current_term: 1 leader_uuid: "c4607b3be6854d73853b8976e4797024" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4607b3be6854d73853b8976e4797024" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 37997 } } }
I20260924 16:13:14.271158 6666 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4607b3be6854d73853b8976e4797024 [sys.catalog]: This master's current role is: LEADER
I20260924 16:13:14.272657 6665 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4607b3be6854d73853b8976e4797024 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4607b3be6854d73853b8976e4797024" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4607b3be6854d73853b8976e4797024" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 37997 } } }
I20260924 16:13:14.272970 6665 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4607b3be6854d73853b8976e4797024 [sys.catalog]: This master's current role is: LEADER
I20260924 16:13:14.273064 6669 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:13:14.279219 6669 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:13:14.288887 6669 catalog_manager.cc:1347] Generated new cluster ID: 9f6aa108138542ef917a5facaccab9bc
I20260924 16:13:14.289019 6669 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:13:14.317529 6669 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:13:14.319001 6669 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:13:14.346725 6669 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4607b3be6854d73853b8976e4797024: Generated new TSK 0
I20260924 16:13:14.347831 6669 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.234.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab.
Entry for principal kudu/127.29.234.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab.
I20260924 16:13:14.408674 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.1:0
--local_ip_for_outbound_sockets=127.29.234.1
--webserver_interface=127.29.234.1
--webserver_port=0
--tserver_master_addrs=127.29.234.62:37997
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab
--principal=kudu/127.29.234.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:44393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:13:14.750639 6685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:13:14.750932 6685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:13:14.751024 6685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:13:14.758445 6685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:13:14.758558 6685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:13:14.758669 6685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.1
I20260924 16:13:14.768837 6685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:44393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab
--principal=kudu/127.29.234.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.234.1
--webserver_port=0
--tserver_master_addrs=127.29.234.62:37997
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.1
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:13:14.770447 6685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:13:14.771816 6685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:13:14.785681 6690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:13:14.785552 6693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:13:14.785579 6691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:13:14.786638 6685 server_base.cc:1034] running on GCE node
I20260924 16:13:14.787348 6685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:13:14.788350 6685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:13:14.789584 6685 hybrid_clock.cc:648] HybridClock initialized: now 1790266394789523 us; error 53 us; skew 500 ppm
I20260924 16:13:14.789927 6685 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:13:14 dist-test-slave-v5j9 krb5kdc[6578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266394, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:13:14.793128 6685 init.cc:377] Logged in from keytab as kudu/127.29.234.1@KRBTEST.COM (short username kudu)
I20260924 16:13:14.794796 6685 webserver.cc:460] Webserver started at http://127.29.234.1:40595/ using document root <none> and password file <none>
I20260924 16:13:14.795565 6685 fs_manager.cc:359] Metadata directory not provided
I20260924 16:13:14.795715 6685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:13:14.796043 6685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:13:14.799165 6685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-0/data/instance:
uuid: "ca3a9dfb170148028704b5afe0f99aba"
format_stamp: "Formatted at 2026-09-24 16:13:14 on dist-test-slave-v5j9"
server_key: "f7b89da7ef08baa34ba61584ae7e4202"
server_key_iv: "4c64ea30bab57eeacf1c1f4f37df0dc9"
server_key_version: "encryptionkey@0"
I20260924 16:13:14.800055 6685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-0/wal/instance:
uuid: "ca3a9dfb170148028704b5afe0f99aba"
format_stamp: "Formatted at 2026-09-24 16:13:14 on dist-test-slave-v5j9"
server_key: "f7b89da7ef08baa34ba61584ae7e4202"
server_key_iv: "4c64ea30bab57eeacf1c1f4f37df0dc9"
server_key_version: "encryptionkey@0"
I20260924 16:13:14.805475 6685 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260924 16:13:14.808831 6700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:13:14.810228 6685 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260924 16:13:14.810411 6685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-0/wal
uuid: "ca3a9dfb170148028704b5afe0f99aba"
format_stamp: "Formatted at 2026-09-24 16:13:14 on dist-test-slave-v5j9"
server_key: "f7b89da7ef08baa34ba61584ae7e4202"
server_key_iv: "4c64ea30bab57eeacf1c1f4f37df0dc9"
server_key_version: "encryptionkey@0"
I20260924 16:13:14.810616 6685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:13:14.835364 6685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:13:14.839809 6685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:13:14.840122 6685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:13:14.841142 6685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:13:14.842613 6685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:13:14.842725 6685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:13:14.842885 6685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:13:14.842957 6685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:13:14.877609 6685 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.1:36255
I20260924 16:13:14.877651 6814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.1:36255 every 8 connection(s)
I20260924 16:13:14.879014 6685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
I20260924 16:13:14.881872 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 6685
I20260924 16:13:14.882095 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-0/wal/instance
I20260924 16:13:14.882884 30632 external_mini_cluster.cc:1428] Setting key dd92b78dc5229089618c3fae84546828
Sep 24 16:13:14 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266394, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.1@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:13:14.903546 6817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:14.881750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.1:34065 (local address 127.29.234.62:37997)
0924 16:13:14.882295 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:14.882303 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:14.883540 (+ 1237us) server_negotiation.cc:407] Connection header received
0924 16:13:14.885961 (+ 2421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:14.885966 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:14.886027 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:14.886109 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:14.888297 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:14.889085 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:14.891643 (+ 2558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:14.892306 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:14.892428 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:14.896338 (+ 3910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:13:14.896347 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:13:14.896355 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:13:14.896401 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:13:14.900002 (+ 3601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:14.900822 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:14.900831 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:14.900837 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:14.900998 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:14.901435 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:14.901443 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:14.901449 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:14.901843 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:13:14.901998 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:14.902780 (+ 782us) server_negotiation.cc:299] Negotiation successful
0924 16:13:14.903057 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":256,"threads_started":1}
I20260924 16:13:14.904215 6818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:14.882423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:37997 (local address 127.29.234.1:34065)
0924 16:13:14.883321 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:14.883386 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:14.885641 (+ 2255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:14.886443 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:14.886458 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:14.887285 (+ 827us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:13:14.888130 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:14.888156 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:14.889247 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:14.889255 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:14.891481 (+ 2226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:14.891499 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:14.892540 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:14.892549 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:14.892894 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:14.893685 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:13:14.893709 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:13:14.896082 (+ 2373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:13:14.900181 (+ 4099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:13:14.900190 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:13:14.900206 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:13:14.900667 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:13:14.901109 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:13:14.901112 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:13:14.901114 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:13:14.901291 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:13:14.901965 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:13:14.901971 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:13:14.902334 (+ 363us) client_negotiation.cc:769] Sending connection context
0924 16:13:14.902599 (+ 265us) client_negotiation.cc:240] Negotiation successful
0924 16:13:14.903024 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":313,"threads_started":1}
I20260924 16:13:14.906072 6815 heartbeater.cc:347] Connected to a master server at 127.29.234.62:37997
I20260924 16:13:14.906421 6815 heartbeater.cc:464] Registering TS with master...
I20260924 16:13:14.907173 6815 heartbeater.cc:511] Master 127.29.234.62:37997 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.29.234.2@KRBTEST.COM; defaulting to no policy
I20260924 16:13:14.909483 6624 ts_manager.cc:194] Registered new tserver with Master: ca3a9dfb170148028704b5afe0f99aba (127.29.234.1:36255)
I20260924 16:13:14.911125 6624 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.234.1@KRBTEST.COM'} at 127.29.234.1:34065
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab.
Entry for principal kudu/127.29.234.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab.
I20260924 16:13:14.938176 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.2:0
--local_ip_for_outbound_sockets=127.29.234.2
--webserver_interface=127.29.234.2
--webserver_port=0
--tserver_master_addrs=127.29.234.62:37997
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab
--principal=kudu/127.29.234.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:44393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:13:15.280459 6821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:13:15.280763 6821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:13:15.280859 6821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:13:15.288326 6821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:13:15.288456 6821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:13:15.288573 6821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.2
I20260924 16:13:15.298687 6821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:44393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab
--principal=kudu/127.29.234.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.234.2
--webserver_port=0
--tserver_master_addrs=127.29.234.62:37997
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.2
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:13:15.300078 6821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:13:15.301472 6821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:13:15.315256 6827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:13:15.315217 6829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:13:15.315412 6826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:13:15.316129 6821 server_base.cc:1034] running on GCE node
I20260924 16:13:15.316752 6821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:13:15.317862 6821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:13:15.319126 6821 hybrid_clock.cc:648] HybridClock initialized: now 1790266395319035 us; error 67 us; skew 500 ppm
I20260924 16:13:15.319548 6821 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:13:15 dist-test-slave-v5j9 krb5kdc[6578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266395, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:13:15.322798 6821 init.cc:377] Logged in from keytab as kudu/127.29.234.2@KRBTEST.COM (short username kudu)
I20260924 16:13:15.324394 6821 webserver.cc:460] Webserver started at http://127.29.234.2:37973/ using document root <none> and password file <none>
I20260924 16:13:15.325140 6821 fs_manager.cc:359] Metadata directory not provided
I20260924 16:13:15.325285 6821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:13:15.325646 6821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:13:15.328590 6821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-1/data/instance:
uuid: "85f7a82e0a8b46f5ad203bfec5401868"
format_stamp: "Formatted at 2026-09-24 16:13:15 on dist-test-slave-v5j9"
server_key: "8e5783be4e0abb9ccedb886aef9e7aff"
server_key_iv: "592b38eeebc4e558dabab829358137d9"
server_key_version: "encryptionkey@0"
I20260924 16:13:15.329452 6821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-1/wal/instance:
uuid: "85f7a82e0a8b46f5ad203bfec5401868"
format_stamp: "Formatted at 2026-09-24 16:13:15 on dist-test-slave-v5j9"
server_key: "8e5783be4e0abb9ccedb886aef9e7aff"
server_key_iv: "592b38eeebc4e558dabab829358137d9"
server_key_version: "encryptionkey@0"
I20260924 16:13:15.334690 6821 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260924 16:13:15.338331 6836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:13:15.339572 6821 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 16:13:15.339757 6821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-1/wal
uuid: "85f7a82e0a8b46f5ad203bfec5401868"
format_stamp: "Formatted at 2026-09-24 16:13:15 on dist-test-slave-v5j9"
server_key: "8e5783be4e0abb9ccedb886aef9e7aff"
server_key_iv: "592b38eeebc4e558dabab829358137d9"
server_key_version: "encryptionkey@0"
I20260924 16:13:15.340018 6821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:13:15.354319 6821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:13:15.358505 6821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:13:15.358783 6821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:13:15.359742 6821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:13:15.361119 6821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:13:15.361218 6821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:13:15.361390 6821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:13:15.361466 6821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260924 16:13:15.395550 6821 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.2:44067
I20260924 16:13:15.395550 6951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.2:44067 every 8 connection(s)
I20260924 16:13:15.396911 6821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
I20260924 16:13:15.401381 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 6821
I20260924 16:13:15.401559 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-1/wal/instance
I20260924 16:13:15.402130 30632 external_mini_cluster.cc:1428] Setting key a47da994642091b6e4f1a240c5b450d5
Sep 24 16:13:15 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266395, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.2@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:13:15.422125 6817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:15.399484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.2:54421 (local address 127.29.234.62:37997)
0924 16:13:15.399659 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:15.399664 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:15.400853 (+ 1189us) server_negotiation.cc:407] Connection header received
0924 16:13:15.403365 (+ 2512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:15.403371 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:15.403448 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:15.403556 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:15.405567 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:15.406273 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:15.408737 (+ 2464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:15.409170 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:15.409263 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:15.414340 (+ 5077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:13:15.414360 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:13:15.414367 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:13:15.414424 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:13:15.418255 (+ 3831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:15.419288 (+ 1033us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:15.419297 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:15.419304 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:15.419486 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:15.419954 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:15.419963 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:15.419968 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:15.420396 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:13:15.420561 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:15.421339 (+ 778us) server_negotiation.cc:299] Negotiation successful
0924 16:13:15.421658 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":58}
I20260924 16:13:15.422977 6954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:15.399938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:37997 (local address 127.29.234.2:54421)
0924 16:13:15.400656 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:15.400700 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:15.403040 (+ 2340us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:15.403779 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:15.403788 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:15.404498 (+ 710us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:13:15.405378 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:15.405407 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:15.406429 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:15.406437 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:15.408580 (+ 2143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:15.408599 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:15.409446 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:15.409456 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:15.409827 (+ 371us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:15.410639 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:13:15.410677 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:13:15.414030 (+ 3353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:13:15.418479 (+ 4449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:13:15.418489 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:13:15.418508 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:13:15.419125 (+ 617us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:13:15.419609 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:13:15.419612 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:13:15.419614 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:13:15.419820 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:13:15.420521 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:13:15.420529 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:13:15.420916 (+ 387us) client_negotiation.cc:769] Sending connection context
0924 16:13:15.421161 (+ 245us) client_negotiation.cc:240] Negotiation successful
0924 16:13:15.421706 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":276,"threads_started":1}
I20260924 16:13:15.424693 6952 heartbeater.cc:347] Connected to a master server at 127.29.234.62:37997
I20260924 16:13:15.425060 6952 heartbeater.cc:464] Registering TS with master...
I20260924 16:13:15.426172 6952 heartbeater.cc:511] Master 127.29.234.62:37997 requested a full tablet report, sending...
I20260924 16:13:15.428067 6624 ts_manager.cc:194] Registered new tserver with Master: 85f7a82e0a8b46f5ad203bfec5401868 (127.29.234.2:44067)
WARNING: no policy specified for kudu/127.29.234.3@KRBTEST.COM; defaulting to no policy
I20260924 16:13:15.429392 6624 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.234.2@KRBTEST.COM'} at 127.29.234.2:54421
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab.
Entry for principal kudu/127.29.234.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab.
I20260924 16:13:15.457896 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.3:0
--local_ip_for_outbound_sockets=127.29.234.3
--webserver_interface=127.29.234.3
--webserver_port=0
--tserver_master_addrs=127.29.234.62:37997
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab
--principal=kudu/127.29.234.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:44393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:13:15.798635 6957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:13:15.798936 6957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:13:15.799089 6957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:13:15.806303 6957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:13:15.806433 6957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:13:15.806560 6957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.3
I20260924 16:13:15.816699 6957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:44393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab
--principal=kudu/127.29.234.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.234.3
--webserver_port=0
--tserver_master_addrs=127.29.234.62:37997
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.3
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:13:15.818055 6957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:13:15.819411 6957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:13:15.833421 6962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:13:15.833323 6963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:13:15.833367 6965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:13:15.834237 6957 server_base.cc:1034] running on GCE node
I20260924 16:13:15.834874 6957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:13:15.835813 6957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:13:15.837065 6957 hybrid_clock.cc:648] HybridClock initialized: now 1790266395837019 us; error 52 us; skew 500 ppm
I20260924 16:13:15.837528 6957 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:13:15 dist-test-slave-v5j9 krb5kdc[6578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266395, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:13:15.842218 6957 init.cc:377] Logged in from keytab as kudu/127.29.234.3@KRBTEST.COM (short username kudu)
I20260924 16:13:15.844213 6957 webserver.cc:460] Webserver started at http://127.29.234.3:34503/ using document root <none> and password file <none>
I20260924 16:13:15.844993 6957 fs_manager.cc:359] Metadata directory not provided
I20260924 16:13:15.845126 6957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:13:15.845531 6957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:13:15.848673 6957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-2/data/instance:
uuid: "fa7e48aee5274e668c2c0984fa54c011"
format_stamp: "Formatted at 2026-09-24 16:13:15 on dist-test-slave-v5j9"
server_key: "5ed0417c91454af86ce317aebf536975"
server_key_iv: "e212074ccf89417f04fba533c1ee7254"
server_key_version: "encryptionkey@0"
I20260924 16:13:15.849582 6957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-2/wal/instance:
uuid: "fa7e48aee5274e668c2c0984fa54c011"
format_stamp: "Formatted at 2026-09-24 16:13:15 on dist-test-slave-v5j9"
server_key: "5ed0417c91454af86ce317aebf536975"
server_key_iv: "e212074ccf89417f04fba533c1ee7254"
server_key_version: "encryptionkey@0"
I20260924 16:13:15.854846 6957 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260924 16:13:15.858436 6972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:13:15.859831 6957 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 16:13:15.860033 6957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-2/wal
uuid: "fa7e48aee5274e668c2c0984fa54c011"
format_stamp: "Formatted at 2026-09-24 16:13:15 on dist-test-slave-v5j9"
server_key: "5ed0417c91454af86ce317aebf536975"
server_key_iv: "e212074ccf89417f04fba533c1ee7254"
server_key_version: "encryptionkey@0"
I20260924 16:13:15.860275 6957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:13:15.874212 6957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:13:15.879142 6957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:13:15.879475 6957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:13:15.880475 6957 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:13:15.882055 6957 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:13:15.882165 6957 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:13:15.882349 6957 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:13:15.882426 6957 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:13:15.914750 6815 heartbeater.cc:503] Master 127.29.234.62:37997 was elected leader, sending a full tablet report...
I20260924 16:13:15.916303 6957 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.3:38891
I20260924 16:13:15.916320 7086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.3:38891 every 8 connection(s)
I20260924 16:13:15.917732 6957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
I20260924 16:13:15.919902 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 6957
I20260924 16:13:15.920109 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-2/wal/instance
I20260924 16:13:15.920627 30632 external_mini_cluster.cc:1428] Setting key 74fa6b56bb6f60d246c93d849579435f
Sep 24 16:13:15 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266395, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.3@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:13:15.938052 6817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:15.920667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.3:37841 (local address 127.29.234.62:37997)
0924 16:13:15.920871 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:15.920877 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:15.922239 (+ 1362us) server_negotiation.cc:407] Connection header received
0924 16:13:15.924473 (+ 2234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:15.924477 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:15.924551 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:15.924635 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:15.926286 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:15.927014 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:15.928549 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:15.928941 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:15.929031 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:15.932434 (+ 3403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:13:15.932439 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:13:15.932442 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:13:15.932472 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:13:15.934840 (+ 2368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:15.935609 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:15.935614 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:15.935617 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:15.935758 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:15.936208 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:15.936217 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:15.936219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:15.936518 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:13:15.936678 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:15.937461 (+ 783us) server_negotiation.cc:299] Negotiation successful
0924 16:13:15.937691 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 16:13:15.939105 7089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:15.921167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:37997 (local address 127.29.234.3:37841)
0924 16:13:15.922068 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:15.922121 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:15.924220 (+ 2099us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:15.924871 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:15.924881 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:15.925521 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:13:15.926114 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:15.926132 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:15.927167 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:15.927172 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:15.928425 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:15.928437 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:15.929122 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:15.929127 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:15.929324 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:15.930097 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:13:15.930119 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:13:15.932226 (+ 2107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:13:15.935013 (+ 2787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:13:15.935023 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:13:15.935040 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:13:15.935470 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:13:15.935874 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:13:15.935879 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:13:15.935883 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:13:15.936082 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:13:15.936664 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:13:15.936673 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:13:15.937041 (+ 368us) client_negotiation.cc:769] Sending connection context
0924 16:13:15.937318 (+ 277us) client_negotiation.cc:240] Negotiation successful
0924 16:13:15.937928 (+ 610us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":389,"threads_started":1}
I20260924 16:13:15.940871 7087 heartbeater.cc:347] Connected to a master server at 127.29.234.62:37997
I20260924 16:13:15.941174 7087 heartbeater.cc:464] Registering TS with master...
I20260924 16:13:15.941968 7087 heartbeater.cc:511] Master 127.29.234.62:37997 requested a full tablet report, sending...
I20260924 16:13:15.943470 6624 ts_manager.cc:194] Registered new tserver with Master: fa7e48aee5274e668c2c0984fa54c011 (127.29.234.3:38891)
I20260924 16:13:15.944360 6624 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.234.3@KRBTEST.COM'} at 127.29.234.3:37841
I20260924 16:13:15.948390 30632 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.29.234.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.234.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.234.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/krb5kdc/oryx_127.29.234.1.keytab.
Entry for principal oryx/127.29.234.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/krb5kdc/oryx_127.29.234.1.keytab.
WARNING: no policy specified for oryx/127.29.234.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.234.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.234.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/krb5kdc/oryx_127.29.234.2.keytab.
Entry for principal oryx/127.29.234.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/krb5kdc/oryx_127.29.234.2.keytab.
WARNING: no policy specified for oryx/127.29.234.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.234.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.234.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/krb5kdc/oryx_127.29.234.3.keytab.
Entry for principal oryx/127.29.234.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/krb5kdc/oryx_127.29.234.3.keytab.
I20260924 16:13:16.090135 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 6685
Sep 24 16:13:16 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.234.1@KRBTEST.COM
I20260924 16:13:16.106290 7097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:16.091276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36208 (local address 127.29.234.1:36255)
0924 16:13:16.091929 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:16.091936 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:16.091964 (+ 28us) server_negotiation.cc:407] Connection header received
0924 16:13:16.092035 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:16.092042 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:16.092297 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:16.092457 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:16.093860 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:16.095019 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:16.096246 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:16.096807 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:16.096906 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:16.100935 (+ 4029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:13:16.100947 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:13:16.100964 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:13:16.101003 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:13:16.103703 (+ 2700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:16.104378 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:16.104385 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:16.104388 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:16.104505 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:16.104823 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:16.104826 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:16.104828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:16.105150 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:13:16.105449 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:16.105673 (+ 224us) server_negotiation.cc:299] Negotiation successful
0924 16:13:16.105869 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":328,"threads_started":1}
I20260924 16:13:16.107702 6810 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:13:16.193610 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 6685
I20260924 16:13:16.210700 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 6821
Sep 24 16:13:16 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.234.2@KRBTEST.COM
I20260924 16:13:16.226579 7099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:16.211587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44832 (local address 127.29.234.2:44067)
0924 16:13:16.212205 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:16.212213 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:16.212239 (+ 26us) server_negotiation.cc:407] Connection header received
0924 16:13:16.212306 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:16.212312 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:16.212536 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:16.212672 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:16.213972 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:16.214997 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:16.216248 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:16.216821 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:16.216919 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:16.220935 (+ 4016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:13:16.220948 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:13:16.220964 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:13:16.220998 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:13:16.223765 (+ 2767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:16.224357 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:16.224364 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:16.224366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:16.224483 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:16.224819 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:16.224823 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:16.224826 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:16.225258 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:13:16.225584 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:16.225878 (+ 294us) server_negotiation.cc:299] Negotiation successful
0924 16:13:16.226177 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":334,"threads_started":1}
I20260924 16:13:16.228166 6947 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:13:16.309906 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 6821
I20260924 16:13:16.326640 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 6957
Sep 24 16:13:16 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.234.3@KRBTEST.COM
I20260924 16:13:16.342691 7102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:16.327574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49348 (local address 127.29.234.3:38891)
0924 16:13:16.328259 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:16.328266 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:16.328298 (+ 32us) server_negotiation.cc:407] Connection header received
0924 16:13:16.328362 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:16.328368 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:16.328587 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:16.328718 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:16.330023 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:16.331041 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:16.332217 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:16.332855 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:16.332949 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:16.337227 (+ 4278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:13:16.337238 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:13:16.337275 (+ 37us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:13:16.337307 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:13:16.339957 (+ 2650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:16.340549 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:16.340556 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:16.340558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:16.340674 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:16.340966 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:16.340969 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:16.340971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:16.341306 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:13:16.341748 (+ 442us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:16.342024 (+ 276us) server_negotiation.cc:299] Negotiation successful
0924 16:13:16.342341 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":344,"threads_started":1}
I20260924 16:13:16.344106 7081 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:13:16.427189 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 6957
I20260924 16:13:16.444036 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 6591
I20260924 16:13:16.445194 6654 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:13:16.550143 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 6591
I20260924 16:13:16.571252 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:37997
--webserver_interface=127.29.234.62
--webserver_port=41429
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab
--principal=kudu/127.29.234.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:44393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.234.62:37997
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:13:16.909502 7105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:13:16.909824 7105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:13:16.910013 7105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:13:16.917290 7105 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:13:16.917495 7105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:13:16.917609 7105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:13:16.917692 7105 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:13:16.917759 7105 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:13:16.928854 7105 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:44393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.234.62:37997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab
--principal=kudu/127.29.234.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:37997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.234.62
--webserver_port=41429
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:13:16.930370 7105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:13:16.931833 7105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:13:16.944721 7110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:13:16.944758 7111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:13:16.944687 7113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:13:16.945654 7105 server_base.cc:1034] running on GCE node
I20260924 16:13:16.946556 7105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:13:16.948021 7105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:13:16.949263 7105 hybrid_clock.cc:648] HybridClock initialized: now 1790266396949201 us; error 48 us; skew 500 ppm
I20260924 16:13:16.949643 7105 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:13:16 dist-test-slave-v5j9 krb5kdc[6578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266396, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:13:16.952855 7105 init.cc:377] Logged in from keytab as kudu/127.29.234.62@KRBTEST.COM (short username kudu)
I20260924 16:13:16.954384 7105 webserver.cc:460] Webserver started at http://127.29.234.62:41429/ using document root <none> and password file <none>
I20260924 16:13:16.955114 7105 fs_manager.cc:359] Metadata directory not provided
I20260924 16:13:16.955274 7105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:13:16.960634 7105 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260924 16:13:16.963493 7120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:13:16.964779 7105 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 16:13:16.964970 7105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/master-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/master-0/wal
uuid: "c4607b3be6854d73853b8976e4797024"
format_stamp: "Formatted at 2026-09-24 16:13:14 on dist-test-slave-v5j9"
server_key: "efb0f67c24860fb8e4b78709e1adabc2"
server_key_iv: "afb603754911af58e8acc24dd6b36439"
server_key_version: "encryptionkey@0"
I20260924 16:13:16.966050 7105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:13:16.985312 7105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:13:16.989511 7105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:13:16.989789 7105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:13:17.008808 7105 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.62:37997
I20260924 16:13:17.008828 7173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.62:37997 every 8 connection(s)
I20260924 16:13:17.010432 7105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
I20260924 16:13:17.014113 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 7105
I20260924 16:13:17.014942 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.1:36255
--local_ip_for_outbound_sockets=127.29.234.1
--tserver_master_addrs=127.29.234.62:37997
--webserver_port=40595
--webserver_interface=127.29.234.1
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab
--principal=kudu/127.29.234.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:44393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.29.234.1
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/krb5kdc/oryx_127.29.234.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:13:17.017695 7174 sys_catalog.cc:263] Verifying existing consensus state
I20260924 16:13:17.020967 7174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4607b3be6854d73853b8976e4797024: Bootstrap starting.
I20260924 16:13:17.040143 7174 log.cc:824] T 00000000000000000000000000000000 P c4607b3be6854d73853b8976e4797024: Log is configured to *not* fsync() on all Append() calls
I20260924 16:13:17.049086 7174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4607b3be6854d73853b8976e4797024: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 16:13:17.049698 7174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4607b3be6854d73853b8976e4797024: Bootstrap complete.
I20260924 16:13:17.058939 7174 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4607b3be6854d73853b8976e4797024 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4607b3be6854d73853b8976e4797024" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 37997 } }
I20260924 16:13:17.059851 7174 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4607b3be6854d73853b8976e4797024 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4607b3be6854d73853b8976e4797024, State: Initialized, Role: FOLLOWER
I20260924 16:13:17.060592 7174 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4607b3be6854d73853b8976e4797024 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4607b3be6854d73853b8976e4797024" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 37997 } }
I20260924 16:13:17.060842 7174 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4607b3be6854d73853b8976e4797024 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:13:17.060976 7174 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4607b3be6854d73853b8976e4797024 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:13:17.061151 7174 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4607b3be6854d73853b8976e4797024 [term 1 FOLLOWER]: Advancing to term 2
I20260924 16:13:17.065040 7174 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4607b3be6854d73853b8976e4797024 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4607b3be6854d73853b8976e4797024" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 37997 } }
I20260924 16:13:17.065625 7174 leader_election.cc:302] T 00000000000000000000000000000000 P c4607b3be6854d73853b8976e4797024 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4607b3be6854d73853b8976e4797024; no voters:
I20260924 16:13:17.066293 7174 leader_election.cc:288] T 00000000000000000000000000000000 P c4607b3be6854d73853b8976e4797024 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 16:13:17.066430 7178 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4607b3be6854d73853b8976e4797024 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 16:13:17.067588 7178 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4607b3be6854d73853b8976e4797024 [term 2 LEADER]: Becoming Leader. State: Replica: c4607b3be6854d73853b8976e4797024, State: Running, Role: LEADER
I20260924 16:13:17.068241 7178 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4607b3be6854d73853b8976e4797024 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4607b3be6854d73853b8976e4797024" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 37997 } }
I20260924 16:13:17.068641 7174 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4607b3be6854d73853b8976e4797024 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:13:17.070824 7180 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4607b3be6854d73853b8976e4797024 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4607b3be6854d73853b8976e4797024. Latest consensus state: current_term: 2 leader_uuid: "c4607b3be6854d73853b8976e4797024" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4607b3be6854d73853b8976e4797024" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 37997 } } }
I20260924 16:13:17.071310 7180 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4607b3be6854d73853b8976e4797024 [sys.catalog]: This master's current role is: LEADER
I20260924 16:13:17.071354 7179 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4607b3be6854d73853b8976e4797024 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c4607b3be6854d73853b8976e4797024" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4607b3be6854d73853b8976e4797024" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 37997 } } }
I20260924 16:13:17.071717 7179 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4607b3be6854d73853b8976e4797024 [sys.catalog]: This master's current role is: LEADER
I20260924 16:13:17.077054 7183 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:13:17.084381 7183 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:13:17.086333 7183 catalog_manager.cc:1259] Loaded cluster ID: 9f6aa108138542ef917a5facaccab9bc
I20260924 16:13:17.086503 7183 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:13:17.091789 7183 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:13:17.093833 7183 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c4607b3be6854d73853b8976e4797024: Loaded TSK: 0
I20260924 16:13:17.095664 7183 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 16:13:17.381464 7176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:13:17.381830 7176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:13:17.382012 7176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:13:17.389643 7176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:13:17.389802 7176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:13:17.389981 7176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.1
I20260924 16:13:17.400795 7176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:44393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/krb5kdc/oryx_127.29.234.1.keytab
--principal=oryx/127.29.234.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.1:36255
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.234.1
--webserver_port=40595
--tserver_master_addrs=127.29.234.62:37997
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.1
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:13:17.402320 7176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:13:17.403664 7176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:13:17.418658 7201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:13:17.418754 7202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:13:17.418800 7204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:13:17.420224 7176 server_base.cc:1034] running on GCE node
I20260924 16:13:17.420948 7176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:13:17.422029 7176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:13:17.423250 7176 hybrid_clock.cc:648] HybridClock initialized: now 1790266397423180 us; error 61 us; skew 500 ppm
I20260924 16:13:17.423583 7176 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:13:17 dist-test-slave-v5j9 krb5kdc[6578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266397, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.234.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:13:17.426936 7176 init.cc:377] Logged in from keytab as oryx/127.29.234.1@KRBTEST.COM (short username oryx)
I20260924 16:13:17.428582 7176 webserver.cc:460] Webserver started at http://127.29.234.1:40595/ using document root <none> and password file <none>
I20260924 16:13:17.429322 7176 fs_manager.cc:359] Metadata directory not provided
I20260924 16:13:17.429531 7176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:13:17.435424 7176 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 16:13:17.438457 7212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:13:17.439981 7176 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:13:17.440161 7176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-0/wal
uuid: "ca3a9dfb170148028704b5afe0f99aba"
format_stamp: "Formatted at 2026-09-24 16:13:14 on dist-test-slave-v5j9"
server_key: "f7b89da7ef08baa34ba61584ae7e4202"
server_key_iv: "4c64ea30bab57eeacf1c1f4f37df0dc9"
server_key_version: "encryptionkey@0"
I20260924 16:13:17.441149 7176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:13:17.457199 7176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:13:17.462064 7176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:13:17.462352 7176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:13:17.463486 7176 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:13:17.464982 7176 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:13:17.465102 7176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:13:17.465226 7176 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:13:17.465317 7176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:13:17.499343 7176 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.1:36255
I20260924 16:13:17.499439 7326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.1:36255 every 8 connection(s)
I20260924 16:13:17.500907 7176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
I20260924 16:13:17.508823 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 7176
I20260924 16:13:17.509559 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.2:44067
--local_ip_for_outbound_sockets=127.29.234.2
--tserver_master_addrs=127.29.234.62:37997
--webserver_port=37973
--webserver_interface=127.29.234.2
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab
--principal=kudu/127.29.234.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:44393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.29.234.2
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/krb5kdc/oryx_127.29.234.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 16:13:17 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.1@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:13:17 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.1@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 16:13:17.863196 7331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:13:17.863548 7331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:13:17.863706 7331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:13:17.871587 7331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:13:17.871742 7331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:13:17.871935 7331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.2
I20260924 16:13:17.882046 7331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:44393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/krb5kdc/oryx_127.29.234.2.keytab
--principal=oryx/127.29.234.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.2:44067
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.234.2
--webserver_port=37973
--tserver_master_addrs=127.29.234.62:37997
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.2
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:13:17.883455 7331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:13:17.884934 7331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:13:17.904192 7340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:13:17.904168 7338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:13:17.904358 7337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:13:17.904969 7331 server_base.cc:1034] running on GCE node
I20260924 16:13:17.905835 7331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:13:17.906992 7331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:13:17.908505 7331 hybrid_clock.cc:648] HybridClock initialized: now 1790266397908431 us; error 81 us; skew 500 ppm
I20260924 16:13:17.908860 7331 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:13:17 dist-test-slave-v5j9 krb5kdc[6578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266397, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.234.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:13:17.912119 7331 init.cc:377] Logged in from keytab as oryx/127.29.234.2@KRBTEST.COM (short username oryx)
I20260924 16:13:17.913793 7331 webserver.cc:460] Webserver started at http://127.29.234.2:37973/ using document root <none> and password file <none>
I20260924 16:13:17.914515 7331 fs_manager.cc:359] Metadata directory not provided
I20260924 16:13:17.914649 7331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:13:17.920657 7331 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 16:13:17.923903 7347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:13:17.925786 7331 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260924 16:13:17.925983 7331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-1/wal
uuid: "85f7a82e0a8b46f5ad203bfec5401868"
format_stamp: "Formatted at 2026-09-24 16:13:15 on dist-test-slave-v5j9"
server_key: "8e5783be4e0abb9ccedb886aef9e7aff"
server_key_iv: "592b38eeebc4e558dabab829358137d9"
server_key_version: "encryptionkey@0"
I20260924 16:13:17.926947 7331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:13:17.946269 7331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:13:17.950714 7331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:13:17.951014 7331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:13:17.952118 7331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:13:17.953714 7331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:13:17.953840 7331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:13:17.953984 7331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:13:17.954069 7331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:13:18.001567 7331 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.2:44067
I20260924 16:13:18.001610 7461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.2:44067 every 8 connection(s)
I20260924 16:13:18.003130 7331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
I20260924 16:13:18.008885 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 7331
I20260924 16:13:18.009680 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.3:38891
--local_ip_for_outbound_sockets=127.29.234.3
--tserver_master_addrs=127.29.234.62:37997
--webserver_port=34503
--webserver_interface=127.29.234.3
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab
--principal=kudu/127.29.234.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:44393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.29.234.3
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/krb5kdc/oryx_127.29.234.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 16:13:18 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.2@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:13:18 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.2@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 16:13:18.688081 7466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:13:18.688712 7466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:13:18.688956 7466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:13:18.708530 7466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:13:18.708761 7466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:13:18.709066 7466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.3
W20260924 16:13:18.715795 7330 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.201s user 0.004s sys 0.009s
I20260924 16:13:18.717986 7330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:17.504135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:37997 (local address 127.29.234.1:45355)
0924 16:13:17.504990 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:17.505031 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:17.506847 (+ 1816us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:17.508298 (+ 1451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:17.508307 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:17.508859 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:13:17.509580 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:17.509602 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:17.511308 (+ 1706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:17.511314 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:17.512572 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:17.512585 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:17.513755 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:17.513762 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:17.514042 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:17.514769 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:13:17.514792 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:13:18.716639 (+1201847us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.234.62:37997: Server oryx/127.29.234.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":383,"threads_started":1}
W20260924 16:13:18.717985 7329 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:13:17.504129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.1:45355 (local address 127.29.234.62:37997)
0924 16:13:17.504907 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:17.504924 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:17.505181 (+ 257us) server_negotiation.cc:407] Connection header received
0924 16:13:17.507427 (+ 2246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:17.507441 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:17.507807 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:17.508118 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:17.509797 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:17.511114 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:17.512729 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:17.513505 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:17.513711 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:18.717087 (+1203376us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.234.1:45355: BlockingRecv error: recv got EOF from 127.29.234.1:45355 (error 108)
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":279,"threads_started":1}
W20260924 16:13:18.719408 7327 heartbeater.cc:644] Failed to heartbeat to 127.29.234.62:37997 (0 consecutive failures): Not authorized: Failed to ping master at 127.29.234.62:37997: Client connection negotiation failed: client connection to 127.29.234.62:37997: Server oryx/127.29.234.62@KRBTEST.COM not found in Kerberos database
Sep 24 16:13:18 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.1@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:13:18 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.1@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:13:18.729115 7330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:18.720187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:37997 (local address 127.29.234.1:43881)
0924 16:13:18.720388 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:18.720411 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:18.720556 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:18.721099 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:18.721103 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:18.721549 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:13:18.721715 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:18.721723 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:18.722720 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:18.722728 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:18.723689 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:18.723700 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:18.724398 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:18.724404 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:18.724569 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:18.725192 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:13:18.725215 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:13:18.728803 (+ 3588us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.234.62:37997: Server oryx/127.29.234.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":61}
W20260924 16:13:18.729229 7329 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:13:18.720573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.1:43881 (local address 127.29.234.62:37997)
0924 16:13:18.720828 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:18.720832 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:18.720846 (+ 14us) server_negotiation.cc:407] Connection header received
0924 16:13:18.720900 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:18.720904 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:18.720960 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:18.721070 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:18.721859 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:18.722579 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:18.723831 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:18.724224 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:18.724360 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:18.728995 (+ 4635us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.234.1:43881: BlockingRecv error: recv got EOF from 127.29.234.1:43881 (error 108)
Metrics: {"server-negotiator.queue_time_us":162}
I20260924 16:13:18.734650 7466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:44393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/krb5kdc/oryx_127.29.234.3.keytab
--principal=oryx/127.29.234.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.3:38891
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.234.3
--webserver_port=34503
--tserver_master_addrs=127.29.234.62:37997
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.3
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
Sep 24 16:13:18 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.1@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:13:18.737126 7466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
Sep 24 16:13:18 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.1@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:13:18.738785 7330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:18.730037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:37997 (local address 127.29.234.1:52035)
0924 16:13:18.730519 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:18.730533 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:18.730662 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:18.730979 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:18.730984 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:18.731299 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:13:18.731560 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:18.731569 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:18.732539 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:18.732544 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:18.733543 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:18.733555 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:18.734238 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:18.734244 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:18.734414 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:18.735194 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:13:18.735218 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:13:18.738497 (+ 3279us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.234.62:37997: Server oryx/127.29.234.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":399}
W20260924 16:13:18.738941 7329 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:13:18.730127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.1:52035 (local address 127.29.234.62:37997)
0924 16:13:18.730419 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:18.730423 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:18.730605 (+ 182us) server_negotiation.cc:407] Connection header received
0924 16:13:18.730786 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:18.730791 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:18.730865 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:18.730949 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:18.731695 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:18.732398 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:18.733676 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:18.734080 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:18.734214 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:18.738700 (+ 4486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.234.1:52035: BlockingRecv error: recv got EOF from 127.29.234.1:52035 (error 108)
Metrics: {"server-negotiator.queue_time_us":184}
W20260924 16:13:18.739307 7327 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 16:13:18.739698 7466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:13:18.759474 7473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:13:18.759621 7472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:13:18.759536 7475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:13:18.760360 7466 server_base.cc:1034] running on GCE node
I20260924 16:13:18.761039 7466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:13:18.762183 7466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:13:18.763602 7466 hybrid_clock.cc:648] HybridClock initialized: now 1790266398763529 us; error 74 us; skew 500 ppm
I20260924 16:13:18.763980 7466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:13:18 dist-test-slave-v5j9 krb5kdc[6578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266398, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.234.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:13:18.767231 7466 init.cc:377] Logged in from keytab as oryx/127.29.234.3@KRBTEST.COM (short username oryx)
I20260924 16:13:18.768972 7466 webserver.cc:460] Webserver started at http://127.29.234.3:34503/ using document root <none> and password file <none>
I20260924 16:13:18.769908 7466 fs_manager.cc:359] Metadata directory not provided
I20260924 16:13:18.770068 7466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:13:18.775616 7466 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260924 16:13:18.778690 7482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:13:18.780407 7466 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260924 16:13:18.780598 7466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-2/wal
uuid: "fa7e48aee5274e668c2c0984fa54c011"
format_stamp: "Formatted at 2026-09-24 16:13:15 on dist-test-slave-v5j9"
server_key: "5ed0417c91454af86ce317aebf536975"
server_key_iv: "e212074ccf89417f04fba533c1ee7254"
server_key_version: "encryptionkey@0"
I20260924 16:13:18.781647 7466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:13:18.796160 7466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:13:18.800779 7466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:13:18.801050 7466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:13:18.802179 7466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:13:18.803591 7466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:13:18.803710 7466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:13:18.803836 7466 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:13:18.803918 7466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:13:18.849287 7466 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.3:38891
I20260924 16:13:18.849320 7595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.3:38891 every 8 connection(s)
I20260924 16:13:18.850862 7466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
I20260924 16:13:18.860577 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 7466
Sep 24 16:13:18 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.3@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:13:18 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.3@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:13:18.899375 7600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:18.861963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38204 (local address 127.29.234.62:37997)
0924 16:13:18.868456 (+ 6493us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:18.868464 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:18.869426 (+ 962us) server_negotiation.cc:407] Connection header received
0924 16:13:18.869514 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:18.869523 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:18.869632 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:18.869823 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:18.875572 (+ 5749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:18.877460 (+ 1888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:18.879757 (+ 2297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:18.880779 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:18.880959 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:18.886067 (+ 5108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:13:18.886085 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:13:18.886811 (+ 726us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:13:18.886903 (+ 92us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:13:18.892943 (+ 6040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:18.894294 (+ 1351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:18.894311 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:18.894330 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:18.894662 (+ 332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:18.895357 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:18.895366 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:18.895372 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:18.896364 (+ 992us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:13:18.896836 (+ 472us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:18.898102 (+ 1266us) server_negotiation.cc:299] Negotiation successful
0924 16:13:18.898722 (+ 620us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6358,"thread_start_us":6191,"threads_started":1}
W20260924 16:13:19.289857 7465 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.259s user 0.004s sys 0.024s
W20260924 16:13:19.291639 7464 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:13:18.006218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.2:53663 (local address 127.29.234.62:37997)
0924 16:13:18.006673 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:18.006678 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:18.013405 (+ 6727us) server_negotiation.cc:407] Connection header received
0924 16:13:18.018034 (+ 4629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:18.018045 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:18.018182 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:18.018300 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:18.021979 (+ 3679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:18.023869 (+ 1890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:18.027711 (+ 3842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:18.028506 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:18.028670 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:19.291379 (+1262709us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.234.2:53663: BlockingRecv error: recv got EOF from 127.29.234.2:53663 (error 108)
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":242,"threads_started":1}
I20260924 16:13:19.293064 7465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:18.007286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:37997 (local address 127.29.234.2:53663)
0924 16:13:18.008280 (+ 994us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:18.008331 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:18.017486 (+ 9155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:18.019238 (+ 1752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:18.019259 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:18.020418 (+ 1159us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:13:18.021635 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:18.021673 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:18.024219 (+ 2546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:18.024227 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:18.027474 (+ 3247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:18.027501 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:18.028865 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:18.028875 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:18.029451 (+ 576us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:18.030578 (+ 1127us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:13:18.030630 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:13:19.291222 (+1260592us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.234.62:37997: Server oryx/127.29.234.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":642,"thread_start_us":388,"threads_started":1}
W20260924 16:13:19.294245 7462 heartbeater.cc:644] Failed to heartbeat to 127.29.234.62:37997 (0 consecutive failures): Not authorized: Failed to ping master at 127.29.234.62:37997: Client connection negotiation failed: client connection to 127.29.234.62:37997: Server oryx/127.29.234.62@KRBTEST.COM not found in Kerberos database
Sep 24 16:13:19 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.2@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:13:19 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.2@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:13:19.316473 7465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:19.295062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:37997 (local address 127.29.234.2:54079)
0924 16:13:19.295262 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:19.295278 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:19.295399 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:19.296095 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:19.296105 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:19.296796 (+ 691us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:13:19.297114 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:19.297131 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:19.298445 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:19.298454 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:19.300869 (+ 2415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:19.300894 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:19.301813 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:19.301824 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:19.302103 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:19.311266 (+ 9163us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:13:19.311306 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:13:19.316175 (+ 4869us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.234.62:37997: Server oryx/127.29.234.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":74}
W20260924 16:13:19.317005 7464 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:13:19.295594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.2:54079 (local address 127.29.234.62:37997)
0924 16:13:19.295761 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:19.295769 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:19.295788 (+ 19us) server_negotiation.cc:407] Connection header received
0924 16:13:19.295839 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:19.295844 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:19.295918 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:19.295991 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:19.297287 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:19.298277 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:19.301073 (+ 2796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:19.301574 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:19.301673 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:19.316608 (+ 14935us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.234.2:54079: BlockingRecv error: recv got EOF from 127.29.234.2:54079 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 24 16:13:19 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.2@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:13:19 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.2@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 16:13:19.330539 7464 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:13:19.317977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.2:34839 (local address 127.29.234.62:37997)
0924 16:13:19.318249 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:19.318259 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:19.318758 (+ 499us) server_negotiation.cc:407] Connection header received
0924 16:13:19.318989 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:19.318998 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:19.319116 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:19.319245 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:19.319999 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:19.321800 (+ 1801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:19.323239 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:19.324233 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:19.324405 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:19.330049 (+ 5644us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.234.2:34839: BlockingRecv error: recv got EOF from 127.29.234.2:34839 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
I20260924 16:13:19.331426 7465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:19.317416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:37997 (local address 127.29.234.2:34839)
0924 16:13:19.318637 (+ 1221us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:19.318662 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:19.318812 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:19.319369 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:19.319373 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:19.319675 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:13:19.319835 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:19.319844 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:19.322009 (+ 2165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:19.322013 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:19.323064 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:19.323078 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:19.324419 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:19.324423 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:19.324570 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:19.325336 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:13:19.325410 (+ 74us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:13:19.330974 (+ 5564us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.234.62:37997: Server oryx/127.29.234.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1082,"thread_start_us":1630,"threads_started":1}
W20260924 16:13:19.331871 7462 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 24 16:13:19 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.1@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:13:19 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.1@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:13:19.754451 7603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:19.739973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:37997 (local address 127.29.234.1:51497)
0924 16:13:19.740580 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:19.740598 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:19.740708 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:19.741178 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:19.741182 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:19.741638 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:13:19.741784 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:19.741792 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:19.743747 (+ 1955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:19.743751 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:19.744838 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:19.744875 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:19.746514 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:19.746529 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:19.746822 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:19.747924 (+ 1102us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:13:19.747960 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:13:19.754067 (+ 6107us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.234.62:37997: Server oryx/127.29.234.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":380,"threads_started":1}
W20260924 16:13:19.754879 7464 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:13:19.740109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.1:51497 (local address 127.29.234.62:37997)
0924 16:13:19.740349 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:19.740360 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:19.740700 (+ 340us) server_negotiation.cc:407] Connection header received
0924 16:13:19.740860 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:19.740869 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:19.741022 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:19.741166 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:19.741947 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:19.743563 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:19.745094 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:19.746102 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:19.746289 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:19.754497 (+ 8208us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.234.1:51497: BlockingRecv error: recv got EOF from 127.29.234.1:51497 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260924 16:13:19.970141 7598 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.104s user 0.004s sys 0.010s
W20260924 16:13:19.971334 7329 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:13:18.853946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.3:50259 (local address 127.29.234.62:37997)
0924 16:13:18.854114 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:18.854120 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:18.856143 (+ 2023us) server_negotiation.cc:407] Connection header received
0924 16:13:18.858156 (+ 2013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:18.858161 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:18.858296 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:18.858410 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:18.859922 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:18.860627 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:18.863178 (+ 2551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:18.864105 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:18.864238 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:19.971137 (+1106899us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.234.3:50259: BlockingRecv error: recv got EOF from 127.29.234.3:50259 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 16:13:19.971951 7598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:18.855054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:37997 (local address 127.29.234.3:50259)
0924 16:13:18.855978 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:18.856017 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:18.857874 (+ 1857us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:18.858654 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:18.858663 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:18.859170 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:13:18.859737 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:18.859754 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:18.860842 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:18.860845 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:18.862949 (+ 2104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:18.862971 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:18.864659 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:18.864668 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:18.865058 (+ 390us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:18.865962 (+ 904us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:13:18.865991 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:13:19.970958 (+1104967us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.234.62:37997: Server oryx/127.29.234.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":649,"thread_start_us":390,"threads_started":1}
W20260924 16:13:19.972963 7596 heartbeater.cc:644] Failed to heartbeat to 127.29.234.62:37997 (0 consecutive failures): Not authorized: Failed to ping master at 127.29.234.62:37997: Client connection negotiation failed: client connection to 127.29.234.62:37997: Server oryx/127.29.234.62@KRBTEST.COM not found in Kerberos database
Sep 24 16:13:19 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.3@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:13:19 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.3@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:13:19.981807 7598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:19.973678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:37997 (local address 127.29.234.3:55097)
0924 16:13:19.973870 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:19.973889 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:19.974010 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:19.974374 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:19.974379 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:19.974770 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:13:19.974935 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:19.974944 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:19.975960 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:19.975963 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:19.976892 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:19.976902 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:19.977658 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:19.977674 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:19.977787 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:19.978427 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:13:19.978444 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:13:19.981613 (+ 3169us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.234.62:37997: Server oryx/127.29.234.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":58}
W20260924 16:13:19.982007 7329 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:13:19.973699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.3:55097 (local address 127.29.234.62:37997)
0924 16:13:19.973926 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:19.973933 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:19.973949 (+ 16us) server_negotiation.cc:407] Connection header received
0924 16:13:19.974158 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:19.974163 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:19.974227 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:19.974314 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:19.975058 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:19.975845 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:19.977010 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:19.977515 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:19.977607 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:19.981796 (+ 4189us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.234.3:55097: BlockingRecv error: recv got EOF from 127.29.234.3:55097 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Sep 24 16:13:19 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.3@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:13:19 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.3@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:13:19.990722 7598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:19.982490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:37997 (local address 127.29.234.3:48503)
0924 16:13:19.982641 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:19.982658 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:19.982768 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:19.983005 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:19.983008 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:19.983263 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:13:19.983402 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:19.983409 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:19.984342 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:19.984345 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:19.985593 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:19.985610 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:19.986615 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:19.986626 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:19.986745 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:19.987334 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:13:19.987351 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:13:19.990531 (+ 3180us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.234.62:37997: Server oryx/127.29.234.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 16:13:19.991000 7329 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:13:19.982582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.3:48503 (local address 127.29.234.62:37997)
0924 16:13:19.982725 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:19.982729 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:19.982744 (+ 15us) server_negotiation.cc:407] Connection header received
0924 16:13:19.982840 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:19.982843 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:19.982891 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:19.982962 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:19.983511 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:19.984207 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:19.985760 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:19.986387 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:19.986591 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:19.990736 (+ 4145us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.234.3:48503: BlockingRecv error: recv got EOF from 127.29.234.3:48503 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260924 16:13:19.991042 7596 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 24 16:13:20 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.2@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:13:20 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.2@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:13:20.341797 7605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:20.332400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:37997 (local address 127.29.234.2:45151)
0924 16:13:20.332942 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:20.332957 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:20.333135 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:20.333530 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:20.333534 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:20.333990 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:13:20.334135 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:20.334144 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:20.335149 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:20.335155 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:20.336101 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:20.336112 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:20.336824 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:20.336830 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:20.337002 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:20.337717 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:13:20.337735 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:13:20.341570 (+ 3835us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.234.62:37997: Server oryx/127.29.234.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":329,"threads_started":1}
W20260924 16:13:20.342083 7329 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:13:20.332482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.2:45151 (local address 127.29.234.62:37997)
0924 16:13:20.332763 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:20.332771 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:20.333058 (+ 287us) server_negotiation.cc:407] Connection header received
0924 16:13:20.333261 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:20.333266 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:20.333386 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:20.333494 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:20.334286 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:20.335006 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:20.336248 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:20.336643 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:20.336794 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:20.341791 (+ 4997us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.234.2:45151: BlockingRecv error: recv got EOF from 127.29.234.2:45151 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Sep 24 16:13:20 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.1@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:13:20 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.1@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:13:20.764878 7606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:20.755605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:37997 (local address 127.29.234.1:55953)
0924 16:13:20.756290 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:20.756312 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:20.756458 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:20.756817 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:20.756821 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:20.757224 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:13:20.757432 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:20.757442 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:20.758571 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:20.758574 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:20.759507 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:20.759519 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:20.760296 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:20.760300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:20.760439 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:20.761137 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:13:20.761155 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:13:20.764605 (+ 3450us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.234.62:37997: Server oryx/127.29.234.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":404,"threads_started":1}
W20260924 16:13:20.765116 7329 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:13:20.755652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.1:55953 (local address 127.29.234.62:37997)
0924 16:13:20.755896 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:20.755906 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:20.756397 (+ 491us) server_negotiation.cc:407] Connection header received
0924 16:13:20.756590 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:20.756597 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:20.756689 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:20.756771 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:20.757556 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:20.758424 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:20.759657 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:20.760116 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:20.760219 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:20.764838 (+ 4619us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.234.1:55953: BlockingRecv error: recv got EOF from 127.29.234.1:55953 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 24 16:13:20 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.3@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:13:20 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.3@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:13:21.001219 7607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:20.991726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:37997 (local address 127.29.234.3:52485)
0924 16:13:20.992278 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:20.992294 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:20.992454 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:20.992825 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:20.992830 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:20.993311 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:13:20.993528 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:20.993537 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:20.994628 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:20.994644 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:20.995588 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:20.995600 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:20.996347 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:20.996354 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:20.996531 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:20.997139 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:13:20.997162 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:13:21.000973 (+ 3811us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.234.62:37997: Server oryx/127.29.234.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":311,"threads_started":1}
W20260924 16:13:21.001528 7329 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:13:20.991829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.3:52485 (local address 127.29.234.62:37997)
0924 16:13:20.992085 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:20.992094 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:20.992386 (+ 292us) server_negotiation.cc:407] Connection header received
0924 16:13:20.992602 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:20.992607 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:20.992685 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:20.992796 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:20.993678 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:20.994472 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:20.995750 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:20.996154 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:20.996285 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:21.001235 (+ 4950us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.234.3:52485: BlockingRecv error: recv got EOF from 127.29.234.3:52485 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 24 16:13:21 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.2@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:13:21 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.2@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:13:21.352499 7608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:21.342722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:37997 (local address 127.29.234.2:40119)
0924 16:13:21.343225 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:21.343243 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:21.343399 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:21.343752 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:21.343756 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:21.344311 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:13:21.344509 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:21.344519 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:21.345709 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:21.345714 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:21.346698 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:21.346711 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:21.347385 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:21.347390 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:21.347547 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:21.348312 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:13:21.348336 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:13:21.352220 (+ 3884us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.234.62:37997: Server oryx/127.29.234.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":276,"threads_started":1}
W20260924 16:13:21.352593 7329 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:13:21.342827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.2:40119 (local address 127.29.234.62:37997)
0924 16:13:21.343087 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:21.343093 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:21.343358 (+ 265us) server_negotiation.cc:407] Connection header received
0924 16:13:21.343504 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:21.343507 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:21.343573 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:21.343654 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:21.344653 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:21.345562 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:21.346826 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:21.347226 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:21.347316 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:21.352423 (+ 5107us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.234.2:40119: BlockingRecv error: recv got EOF from 127.29.234.2:40119 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
Sep 24 16:13:21 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.1@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:13:21 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.1@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:13:21.775398 7609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:21.765901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:37997 (local address 127.29.234.1:41891)
0924 16:13:21.766436 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:21.766459 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:21.766614 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:21.766939 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:21.766943 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:21.767318 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:13:21.767504 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:21.767513 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:21.768554 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:21.768584 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:21.769654 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:21.769666 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:21.770417 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:21.770423 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:21.770597 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:21.771387 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:13:21.771412 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:13:21.775083 (+ 3671us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.234.62:37997: Server oryx/127.29.234.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":273,"threads_started":1}
W20260924 16:13:21.775581 7329 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:13:21.766123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.1:41891 (local address 127.29.234.62:37997)
0924 16:13:21.766412 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:21.766418 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:21.766547 (+ 129us) server_negotiation.cc:407] Connection header received
0924 16:13:21.766745 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:21.766750 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:21.766808 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:21.766915 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:21.767657 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:21.768399 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:21.769831 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:21.770266 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:21.770403 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:21.775361 (+ 4958us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.234.1:41891: BlockingRecv error: recv got EOF from 127.29.234.1:41891 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 24 16:13:22 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.3@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:13:22 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.3@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:13:22.012216 7610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:22.002438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:37997 (local address 127.29.234.3:33773)
0924 16:13:22.002951 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:22.002966 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:22.003130 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:22.003500 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:22.003504 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:22.003939 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:13:22.004120 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:22.004129 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:22.005139 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:22.005146 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:22.006268 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:22.006280 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:22.007083 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:22.007089 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:22.007272 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:22.007825 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:13:22.007842 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:13:22.011900 (+ 4058us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.234.62:37997: Server oryx/127.29.234.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":308,"threads_started":1}
W20260924 16:13:22.012312 7329 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:13:22.002516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.3:33773 (local address 127.29.234.62:37997)
0924 16:13:22.002712 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:22.002721 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:22.003059 (+ 338us) server_negotiation.cc:407] Connection header received
0924 16:13:22.003268 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:22.003273 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:22.003362 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:22.003468 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:22.004272 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:22.004991 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:22.006409 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:22.006917 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:22.007067 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:22.012071 (+ 5004us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.234.3:33773: BlockingRecv error: recv got EOF from 127.29.234.3:33773 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 24 16:13:22 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.2@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:13:22 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.2@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:13:22.362936 7611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:22.353638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:37997 (local address 127.29.234.2:36453)
0924 16:13:22.354157 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:22.354174 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:22.354287 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:22.354572 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:22.354576 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:22.354918 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:13:22.355141 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:22.355154 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:22.356162 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:22.356166 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:22.357055 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:22.357066 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:22.357943 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:22.357948 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:22.358075 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:22.358717 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:13:22.358735 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:13:22.362576 (+ 3841us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.234.62:37997: Server oryx/127.29.234.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":290,"threads_started":1}
W20260924 16:13:22.363098 7329 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:13:22.353706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.2:36453 (local address 127.29.234.62:37997)
0924 16:13:22.353879 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:22.353883 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:22.354244 (+ 361us) server_negotiation.cc:407] Connection header received
0924 16:13:22.354383 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:22.354386 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:22.354448 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:22.354530 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:22.355290 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:22.356040 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:22.357178 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:22.357820 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:22.357917 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:22.362811 (+ 4894us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.234.2:36453: BlockingRecv error: recv got EOF from 127.29.234.2:36453 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 24 16:13:22 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.1@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:13:22 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.1@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:13:22.784726 7612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:22.776611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:37997 (local address 127.29.234.1:47665)
0924 16:13:22.777108 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:22.777123 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:22.777229 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:22.777570 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:22.777573 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:22.777939 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:13:22.778102 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:22.778111 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:22.779044 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:22.779048 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:22.780026 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:22.780037 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:22.780661 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:22.780665 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:22.780786 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:22.781341 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:13:22.781404 (+ 63us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:13:22.784481 (+ 3077us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.234.62:37997: Server oryx/127.29.234.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":261,"threads_started":1}
W20260924 16:13:22.784893 7329 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:13:22.776657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.1:47665 (local address 127.29.234.62:37997)
0924 16:13:22.776876 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:22.776886 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:22.777204 (+ 318us) server_negotiation.cc:407] Connection header received
0924 16:13:22.777331 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:22.777335 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:22.777441 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:22.777520 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:22.778220 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:22.778939 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:22.780143 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:22.780544 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:22.780631 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:22.784670 (+ 4039us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.234.1:47665: BlockingRecv error: recv got EOF from 127.29.234.1:47665 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 24 16:13:23 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.3@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:13:23 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.3@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:13:23.023155 7613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:23.014143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:37997 (local address 127.29.234.3:60121)
0924 16:13:23.014840 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:23.014859 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:23.015008 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:23.015376 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:23.015381 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:23.015891 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:13:23.016055 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:23.016064 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:23.017057 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:23.017062 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:23.018050 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:23.018062 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:23.018723 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:23.018728 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:23.018909 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:23.019479 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:13:23.019496 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:13:23.022782 (+ 3286us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.234.62:37997: Server oryx/127.29.234.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":450,"threads_started":1}
W20260924 16:13:23.023279 7329 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:13:23.014227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.3:60121 (local address 127.29.234.62:37997)
0924 16:13:23.014461 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:23.014468 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:23.014955 (+ 487us) server_negotiation.cc:407] Connection header received
0924 16:13:23.015135 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:23.015143 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:23.015223 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:23.015330 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:23.016207 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:23.016918 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:23.018192 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:23.018585 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:23.018710 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:23.023011 (+ 4301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.234.3:60121: BlockingRecv error: recv got EOF from 127.29.234.3:60121 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Sep 24 16:13:23 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.2@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:13:23 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.2@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:13:23.374135 7614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:23.364263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:37997 (local address 127.29.234.2:43067)
0924 16:13:23.364959 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:23.364977 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:23.365102 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:23.365548 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:23.365552 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:23.366063 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:13:23.366291 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:23.366306 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:23.367586 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:23.367590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:23.368602 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:23.368615 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:23.369331 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:23.369370 (+ 39us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:23.369553 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:23.370206 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:13:23.370223 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:13:23.373870 (+ 3647us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.234.62:37997: Server oryx/127.29.234.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":399,"threads_started":1}
W20260924 16:13:23.374392 7329 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:13:23.364312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.2:43067 (local address 127.29.234.62:37997)
0924 16:13:23.364495 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:23.364500 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:23.365052 (+ 552us) server_negotiation.cc:407] Connection header received
0924 16:13:23.365266 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:23.365271 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:23.365381 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:23.365498 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:23.366459 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:23.367469 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:23.368736 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:23.369164 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:23.369253 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:23.374123 (+ 4870us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.234.2:43067: BlockingRecv error: recv got EOF from 127.29.234.2:43067 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 24 16:13:23 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.1@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:13:23 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.1@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:13:23.794540 7615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:23.785682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:37997 (local address 127.29.234.1:55957)
0924 16:13:23.786237 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:23.786252 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:23.786424 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:23.786761 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:23.786766 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:23.787228 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:13:23.787385 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:23.787393 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:23.788418 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:23.788425 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:23.789425 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:23.789438 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:23.790125 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:23.790131 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:23.790309 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:23.790959 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:13:23.790976 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:13:23.794299 (+ 3323us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.234.62:37997: Server oryx/127.29.234.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":320,"threads_started":1}
W20260924 16:13:23.794911 7329 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:13:23.785753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.1:55957 (local address 127.29.234.62:37997)
0924 16:13:23.785986 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:23.785994 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:23.786338 (+ 344us) server_negotiation.cc:407] Connection header received
0924 16:13:23.786556 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:23.786561 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:23.786633 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:23.786729 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:23.787526 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:23.788285 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:23.789553 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:23.789986 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:23.790111 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:23.794484 (+ 4373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.234.1:55957: BlockingRecv error: recv got EOF from 127.29.234.1:55957 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 24 16:13:24 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.3@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:13:24 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.3@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:13:24.034053 7616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:24.024465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:37997 (local address 127.29.234.3:47025)
0924 16:13:24.025030 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:24.025046 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:24.025164 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:24.025515 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:24.025518 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:24.025899 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:13:24.026063 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:24.026071 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:24.027100 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:24.027106 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:24.028249 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:24.028262 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:24.029240 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:24.029246 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:24.029495 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:24.030268 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:13:24.030300 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:13:24.033781 (+ 3481us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.234.62:37997: Server oryx/127.29.234.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":331,"threads_started":1}
W20260924 16:13:24.034258 7329 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:13:24.024581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.3:47025 (local address 127.29.234.62:37997)
0924 16:13:24.024880 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:24.024886 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:24.025134 (+ 248us) server_negotiation.cc:407] Connection header received
0924 16:13:24.025274 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:24.025278 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:24.025382 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:24.025463 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:24.026202 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:24.026957 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:24.028408 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:24.028959 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:24.029093 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:24.034009 (+ 4916us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.234.3:47025: BlockingRecv error: recv got EOF from 127.29.234.3:47025 (error 108)
Metrics: {"server-negotiator.queue_time_us":138}
Sep 24 16:13:24 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.2@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:13:24 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.2@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:13:24.393572 7617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:24.375046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:37997 (local address 127.29.234.2:43121)
0924 16:13:24.375566 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:24.375590 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:24.375702 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:24.376041 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:24.376044 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:24.376452 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:13:24.376599 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:24.376607 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:24.377669 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:24.377673 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:24.378611 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:24.378624 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:24.379294 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:24.379298 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:24.379429 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:24.389330 (+ 9901us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:13:24.389405 (+ 75us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:13:24.393222 (+ 3817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.234.62:37997: Server oryx/127.29.234.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":307,"threads_started":1}
W20260924 16:13:24.393819 7329 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:13:24.375230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.2:43121 (local address 127.29.234.62:37997)
0924 16:13:24.375402 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:24.375407 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:24.375659 (+ 252us) server_negotiation.cc:407] Connection header received
0924 16:13:24.375858 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:24.375861 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:24.375924 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:24.376006 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:24.376719 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:24.377541 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:24.378741 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:24.379169 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:24.379291 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:24.393545 (+ 14254us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.234.2:43121: BlockingRecv error: recv got EOF from 127.29.234.2:43121 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 24 16:13:24 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.1@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:13:24 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.1@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:13:24.804236 7618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:24.795591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:37997 (local address 127.29.234.1:43347)
0924 16:13:24.796101 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:24.796125 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:24.796240 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:24.796530 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:24.796533 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:24.796955 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:13:24.797131 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:24.797141 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:24.798215 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:24.798219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:24.799185 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:24.799199 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:24.799945 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:24.799950 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:24.800077 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:24.800686 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:13:24.800703 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:13:24.804005 (+ 3302us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.234.62:37997: Server oryx/127.29.234.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":279,"threads_started":1}
W20260924 16:13:24.804404 7329 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:13:24.795662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.1:43347 (local address 127.29.234.62:37997)
0924 16:13:24.795836 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:24.795841 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:24.796187 (+ 346us) server_negotiation.cc:407] Connection header received
0924 16:13:24.796335 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:24.796338 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:24.796405 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:24.796482 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:24.797270 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:24.798054 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:24.799318 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:24.799754 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:24.799905 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:24.804190 (+ 4285us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.234.1:43347: BlockingRecv error: recv got EOF from 127.29.234.1:43347 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 24 16:13:25 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.3@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:13:25 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.3@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:13:25.044232 7619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:25.035056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:37997 (local address 127.29.234.3:60517)
0924 16:13:25.035615 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:25.035631 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:25.035760 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:25.036095 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:25.036098 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:25.036515 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:13:25.036667 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:25.036675 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:25.037786 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:25.037789 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:25.038762 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:25.038774 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:25.039492 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:25.039496 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:25.039629 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:25.040234 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:13:25.040252 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:13:25.043916 (+ 3664us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.234.62:37997: Server oryx/127.29.234.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":320,"threads_started":1}
W20260924 16:13:25.044449 7329 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:13:25.035154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.3:60517 (local address 127.29.234.62:37997)
0924 16:13:25.035332 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:25.035337 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:25.035706 (+ 369us) server_negotiation.cc:407] Connection header received
0924 16:13:25.035871 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:25.035875 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:25.035953 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:25.036042 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:25.036791 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:25.037652 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:25.038912 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:25.039339 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:25.039442 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:25.044133 (+ 4691us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.234.3:60517: BlockingRecv error: recv got EOF from 127.29.234.3:60517 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 24 16:13:25 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.2@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:13:25 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.2@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:13:25.403682 7620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:25.394704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:37997 (local address 127.29.234.2:33005)
0924 16:13:25.395157 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:25.395173 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:25.395315 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:25.395605 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:25.395608 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:25.396010 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:13:25.396194 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:25.396202 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:25.397243 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:25.397247 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:25.398242 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:25.398264 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:25.398939 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:25.398943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:25.399071 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:25.399721 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:13:25.399739 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:13:25.403350 (+ 3611us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.234.62:37997: Server oryx/127.29.234.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":245,"threads_started":1}
W20260924 16:13:25.403855 7329 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:13:25.394769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.2:33005 (local address 127.29.234.62:37997)
0924 16:13:25.395026 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:25.395034 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:25.395250 (+ 216us) server_negotiation.cc:407] Connection header received
0924 16:13:25.395415 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:25.395419 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:25.395481 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:25.395562 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:25.396369 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:25.397116 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:25.398378 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:25.398785 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:25.398889 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:25.403581 (+ 4692us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.234.2:33005: BlockingRecv error: recv got EOF from 127.29.234.2:33005 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 24 16:13:25 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.1@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:13:25 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.1@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:13:25.814205 7621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:25.805136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:37997 (local address 127.29.234.1:48833)
0924 16:13:25.805723 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:25.805739 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:25.805859 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:25.806175 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:25.806178 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:25.806616 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:13:25.806791 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:25.806800 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:25.807790 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:25.807794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:25.808734 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:25.808746 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:25.809438 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:25.809442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:25.809554 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:25.810119 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:13:25.810136 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:13:25.813814 (+ 3678us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.234.62:37997: Server oryx/127.29.234.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":292,"threads_started":1}
W20260924 16:13:25.814371 7329 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:13:25.805242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.1:48833 (local address 127.29.234.62:37997)
0924 16:13:25.805527 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:25.805533 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:25.805817 (+ 284us) server_negotiation.cc:407] Connection header received
0924 16:13:25.805974 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:25.805977 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:25.806042 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:25.806118 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:25.806918 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:25.807673 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:25.808861 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:25.809269 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:25.809422 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:25.814087 (+ 4665us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.234.1:48833: BlockingRecv error: recv got EOF from 127.29.234.1:48833 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 24 16:13:26 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.3@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:13:26 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.3@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:13:26.055115 7622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:26.045436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:37997 (local address 127.29.234.3:46465)
0924 16:13:26.045932 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:26.045947 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:26.046117 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:26.046556 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:26.046560 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:26.047021 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:13:26.047219 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:26.047228 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:26.048507 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:26.048512 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:26.049726 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:26.049740 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:26.050495 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:26.050502 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:26.050678 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:26.051345 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:13:26.051363 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:13:26.054787 (+ 3424us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.234.62:37997: Server oryx/127.29.234.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":252,"threads_started":1}
W20260924 16:13:26.055254 7329 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:13:26.045495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.3:46465 (local address 127.29.234.62:37997)
0924 16:13:26.045760 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:26.045767 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:26.046044 (+ 277us) server_negotiation.cc:407] Connection header received
0924 16:13:26.046333 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:26.046339 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:26.046418 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:26.046524 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:26.047380 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:26.048368 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:26.049869 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:26.050293 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:26.050456 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:26.055005 (+ 4549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.234.3:46465: BlockingRecv error: recv got EOF from 127.29.234.3:46465 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 24 16:13:26 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.2@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:13:26 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.2@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:13:26.413692 7623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:26.404766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:37997 (local address 127.29.234.2:40503)
0924 16:13:26.405290 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:26.405305 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:26.405493 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:26.405821 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:26.405826 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:26.406286 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:13:26.406455 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:26.406463 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:26.407473 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:26.407480 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:26.408434 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:26.408445 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:26.409155 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:26.409161 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:26.409329 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:26.409973 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:13:26.409990 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:13:26.413288 (+ 3298us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.234.62:37997: Server oryx/127.29.234.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":245,"threads_started":1}
W20260924 16:13:26.413834 7329 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:13:26.404806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.2:40503 (local address 127.29.234.62:37997)
0924 16:13:26.405024 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:26.405034 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:26.405415 (+ 381us) server_negotiation.cc:407] Connection header received
0924 16:13:26.405618 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:26.405624 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:26.405692 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:26.405790 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:26.406602 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:26.407331 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:26.408573 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:26.409004 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:26.409136 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:26.413579 (+ 4443us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.234.2:40503: BlockingRecv error: recv got EOF from 127.29.234.2:40503 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 24 16:13:26 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.1@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:13:26 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.1@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:13:26.825178 7624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:26.815403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:37997 (local address 127.29.234.1:49391)
0924 16:13:26.816056 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:26.816074 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:26.816207 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:26.816690 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:26.816699 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:26.817197 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:13:26.817464 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:26.817476 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:26.818511 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:26.818515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:26.819673 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:26.819691 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:26.820661 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:26.820671 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:26.820871 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:26.821569 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:13:26.821587 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:13:26.824829 (+ 3242us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.234.62:37997: Server oryx/127.29.234.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":361,"threads_started":1}
W20260924 16:13:26.825366 7329 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:13:26.815486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.1:49391 (local address 127.29.234.62:37997)
0924 16:13:26.815710 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:26.815716 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:26.816165 (+ 449us) server_negotiation.cc:407] Connection header received
0924 16:13:26.816355 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:26.816361 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:26.816427 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:26.816513 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:26.817630 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:26.818354 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:26.819864 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:26.820467 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:26.820577 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:26.825083 (+ 4506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.234.1:49391: BlockingRecv error: recv got EOF from 127.29.234.1:49391 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Sep 24 16:13:27 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.3@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:13:27 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.3@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:13:27.066210 7625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:27.056199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:37997 (local address 127.29.234.3:40869)
0924 16:13:27.056879 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:27.056897 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:27.057037 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:27.057411 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:27.057415 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:27.057923 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:13:27.058117 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:27.058127 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:27.059350 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:27.059355 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:27.060374 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:27.060389 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:27.061140 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:27.061144 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:27.061292 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:27.061952 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:13:27.061970 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:13:27.065866 (+ 3896us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.234.62:37997: Server oryx/127.29.234.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":322,"threads_started":1}
W20260924 16:13:27.066378 7329 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:13:27.056371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.3:40869 (local address 127.29.234.62:37997)
0924 16:13:27.056637 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:27.056647 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:27.056994 (+ 347us) server_negotiation.cc:407] Connection header received
0924 16:13:27.057155 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:27.057160 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:27.057226 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:27.057330 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:27.058346 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:27.059147 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:27.060542 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:27.060997 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:27.061100 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:27.066127 (+ 5027us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.234.3:40869: BlockingRecv error: recv got EOF from 127.29.234.3:40869 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 24 16:13:27 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.2@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:13:27 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.2@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:13:27.424288 7626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:27.414887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:37997 (local address 127.29.234.2:54699)
0924 16:13:27.415476 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:27.415492 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:27.415606 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:27.415901 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:27.415905 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:27.416281 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:13:27.416467 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:27.416475 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:27.417688 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:27.417692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:27.419026 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:27.419038 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:27.420043 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:27.420047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:27.420158 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:27.420760 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:13:27.420776 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:13:27.424042 (+ 3266us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.234.62:37997: Server oryx/127.29.234.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":299,"threads_started":1}
W20260924 16:13:27.424459 7329 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:13:27.414976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.2:54699 (local address 127.29.234.62:37997)
0924 16:13:27.415156 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:27.415161 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:27.415564 (+ 403us) server_negotiation.cc:407] Connection header received
0924 16:13:27.415700 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:27.415704 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:27.415773 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:27.415857 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:27.416585 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:27.417577 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:27.419178 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:27.419856 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:27.419978 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:27.424227 (+ 4249us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.234.2:54699: BlockingRecv error: recv got EOF from 127.29.234.2:54699 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 24 16:13:27 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.1@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:13:27 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.1@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:13:27.836115 7627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:27.826379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:37997 (local address 127.29.234.1:56115)
0924 16:13:27.827154 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:27.827170 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:27.827337 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:27.827736 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:27.827742 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:27.828203 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:13:27.828365 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:27.828374 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:27.829461 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:27.829469 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:27.830458 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:27.830471 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:27.831270 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:27.831275 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:27.831432 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:27.832066 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:13:27.832084 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:13:27.835870 (+ 3786us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.234.62:37997: Server oryx/127.29.234.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":669,"thread_start_us":524,"threads_started":1}
W20260924 16:13:27.836371 7329 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:13:27.826466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.1:56115 (local address 127.29.234.62:37997)
0924 16:13:27.826718 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:27.826728 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:27.827272 (+ 544us) server_negotiation.cc:407] Connection header received
0924 16:13:27.827474 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:27.827479 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:27.827587 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:27.827693 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:27.828523 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:27.829281 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:27.830682 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:27.831120 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:27.831230 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:27.836125 (+ 4895us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.234.1:56115: BlockingRecv error: recv got EOF from 127.29.234.1:56115 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 24 16:13:28 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.3@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:13:28 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.3@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:13:28.077159 7628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:28.067369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:37997 (local address 127.29.234.3:50863)
0924 16:13:28.068007 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:28.068023 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:28.068143 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:28.068501 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:28.068505 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:28.068963 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:13:28.069303 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:28.069316 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:28.070558 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:28.070562 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:28.071548 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:28.071560 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:28.072355 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:28.072362 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:28.072488 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:28.073157 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:13:28.073177 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:13:28.076907 (+ 3730us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.234.62:37997: Server oryx/127.29.234.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":367,"threads_started":1}
W20260924 16:13:28.077307 7329 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:13:28.067479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.3:50863 (local address 127.29.234.62:37997)
0924 16:13:28.067711 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:28.067715 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:28.068101 (+ 386us) server_negotiation.cc:407] Connection header received
0924 16:13:28.068243 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:28.068247 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:28.068313 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:28.068420 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:28.069476 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:28.070389 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:28.071707 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:28.072155 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:28.072270 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:28.077106 (+ 4836us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.234.3:50863: BlockingRecv error: recv got EOF from 127.29.234.3:50863 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Sep 24 16:13:28 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.2@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:13:28 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.2@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:13:28.434875 7629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:28.425300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:37997 (local address 127.29.234.2:45679)
0924 16:13:28.425844 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:28.425861 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:28.425973 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:28.426276 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:28.426280 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:28.426693 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:13:28.426856 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:28.426865 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:28.427824 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:28.427827 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:28.428803 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:28.428815 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:28.429503 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:28.429508 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:28.429629 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:28.430340 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:13:28.430370 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:13:28.434522 (+ 4152us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.234.62:37997: Server oryx/127.29.234.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":278,"threads_started":1}
W20260924 16:13:28.435067 7329 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:13:28.425441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.2:45679 (local address 127.29.234.62:37997)
0924 16:13:28.425688 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:28.425695 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:28.425940 (+ 245us) server_negotiation.cc:407] Connection header received
0924 16:13:28.426072 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:28.426076 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:28.426149 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:28.426228 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:28.426980 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:28.427698 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:28.428926 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:28.429383 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:28.429474 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:28.434771 (+ 5297us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.234.2:45679: BlockingRecv error: recv got EOF from 127.29.234.2:45679 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 24 16:13:28 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.1@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:13:28 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.1@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:13:28.846977 7630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:28.837564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:37997 (local address 127.29.234.1:33505)
0924 16:13:28.838103 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:28.838121 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:28.838244 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:28.838602 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:28.838606 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:28.838993 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:13:28.839162 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:28.839171 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:28.840146 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:28.840150 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:28.841175 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:28.841188 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:28.842068 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:28.842073 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:28.842238 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:28.842846 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:13:28.842863 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:13:28.846712 (+ 3849us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.234.62:37997: Server oryx/127.29.234.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":302,"threads_started":1}
W20260924 16:13:28.847122 7329 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:13:28.837620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.1:33505 (local address 127.29.234.62:37997)
0924 16:13:28.837807 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:28.837812 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:28.838224 (+ 412us) server_negotiation.cc:407] Connection header received
0924 16:13:28.838384 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:28.838387 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:28.838456 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:28.838547 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:28.839298 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:28.840019 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:28.841304 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:28.841897 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:28.841999 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:28.846906 (+ 4907us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.234.1:33505: BlockingRecv error: recv got EOF from 127.29.234.1:33505 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 24 16:13:29 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.3@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:13:29 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.3@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:13:29.087805 7631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:29.078341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:37997 (local address 127.29.234.3:58715)
0924 16:13:29.078922 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:29.078943 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:29.079097 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:29.079436 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:29.079439 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:29.080007 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:13:29.080211 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:29.080228 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:29.081268 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:29.081271 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:29.082239 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:29.082251 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:29.082940 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:29.082945 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:29.083077 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:29.083668 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:13:29.083685 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:13:29.087569 (+ 3884us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.234.62:37997: Server oryx/127.29.234.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":385,"threads_started":1}
W20260924 16:13:29.088104 7329 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:13:29.078505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.3:58715 (local address 127.29.234.62:37997)
0924 16:13:29.078727 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:29.078733 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:29.079037 (+ 304us) server_negotiation.cc:407] Connection header received
0924 16:13:29.079218 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:29.079222 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:29.079285 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:29.079384 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:29.080390 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:29.081143 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:29.082354 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:29.082760 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:29.082857 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:29.087791 (+ 4934us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.234.3:58715: BlockingRecv error: recv got EOF from 127.29.234.3:58715 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 24 16:13:29 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.2@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:13:29 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.2@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:13:29.454275 7632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:29.436013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:37997 (local address 127.29.234.2:47581)
0924 16:13:29.436558 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:29.436575 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:29.436693 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:29.437096 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:29.437101 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:29.437572 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:13:29.437737 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:29.437746 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:29.438816 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:29.438824 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:29.439823 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:29.439835 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:29.440597 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:29.440602 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:29.440769 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:29.450059 (+ 9290us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:13:29.450082 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:13:29.453928 (+ 3846us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.234.62:37997: Server oryx/127.29.234.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":315,"threads_started":1}
W20260924 16:13:29.454434 7329 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:13:29.436071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.2:47581 (local address 127.29.234.62:37997)
0924 16:13:29.436292 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:29.436299 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:29.436659 (+ 360us) server_negotiation.cc:407] Connection header received
0924 16:13:29.436839 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:29.436845 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:29.436941 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:29.437057 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:29.437886 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:29.438677 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:29.439980 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:29.440438 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:29.440566 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:29.454181 (+ 13615us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.234.2:47581: BlockingRecv error: recv got EOF from 127.29.234.2:47581 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 24 16:13:29 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.1@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:13:29 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.1@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:13:29.857630 7633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:29.847925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:37997 (local address 127.29.234.1:34791)
0924 16:13:29.848421 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:29.848442 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:29.848566 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:29.848863 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:29.848866 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:29.849274 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:13:29.849546 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:29.849558 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:29.850661 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:29.850668 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:29.852149 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:29.852167 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:29.852924 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:29.852930 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:29.853127 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:29.853839 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:13:29.853868 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:13:29.857308 (+ 3440us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.234.62:37997: Server oryx/127.29.234.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":285,"threads_started":1}
W20260924 16:13:29.857896 7329 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:13:29.848102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.1:34791 (local address 127.29.234.62:37997)
0924 16:13:29.848295 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:29.848299 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:29.848522 (+ 223us) server_negotiation.cc:407] Connection header received
0924 16:13:29.848655 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:29.848658 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:29.848725 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:29.848805 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:29.849703 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:29.850502 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:29.852324 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:29.852745 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:29.852847 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:29.857610 (+ 4763us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.234.1:34791: BlockingRecv error: recv got EOF from 127.29.234.1:34791 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 24 16:13:30 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.3@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:13:30 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.3@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:13:30.098932 7634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:30.088746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:37997 (local address 127.29.234.3:49101)
0924 16:13:30.089392 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:30.089422 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:30.089601 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:30.090034 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:30.090039 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:30.090478 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:13:30.090637 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:30.090647 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:30.091737 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:30.091743 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:30.092758 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:30.092770 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:30.093629 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:30.093634 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:30.093778 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:30.094411 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:13:30.094429 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:13:30.098691 (+ 4262us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.234.62:37997: Server oryx/127.29.234.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":347,"threads_started":1}
W20260924 16:13:30.099094 7329 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:13:30.088947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.3:49101 (local address 127.29.234.62:37997)
0924 16:13:30.089241 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:30.089248 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:30.089517 (+ 269us) server_negotiation.cc:407] Connection header received
0924 16:13:30.089790 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:30.089796 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:30.089877 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:30.089998 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:30.090813 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:30.091576 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:30.092914 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:30.093431 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:30.093594 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:30.098874 (+ 5280us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.234.3:49101: BlockingRecv error: recv got EOF from 127.29.234.3:49101 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
Sep 24 16:13:30 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.2@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:13:30 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.2@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:13:30.464759 7635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:30.455440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:37997 (local address 127.29.234.2:33405)
0924 16:13:30.456025 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:30.456047 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:30.456211 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:30.456649 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:30.456674 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:30.457302 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:13:30.457595 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:30.457605 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:30.458696 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:30.458701 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:30.459677 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:30.459689 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:30.460358 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:30.460363 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:30.460487 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:30.461077 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:13:30.461094 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:13:30.464534 (+ 3440us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.234.62:37997: Server oryx/127.29.234.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":341,"threads_started":1}
W20260924 16:13:30.464998 7329 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:13:30.455597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.2:33405 (local address 127.29.234.62:37997)
0924 16:13:30.455855 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:30.455863 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:30.456148 (+ 285us) server_negotiation.cc:407] Connection header received
0924 16:13:30.456338 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:30.456342 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:30.456447 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:30.456560 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:30.457735 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:30.458504 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:30.459805 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:30.460211 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:30.460304 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:30.464742 (+ 4438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.234.2:33405: BlockingRecv error: recv got EOF from 127.29.234.2:33405 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 24 16:13:30 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.1@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:13:30 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.1@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:13:30.868111 7636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:30.858857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:37997 (local address 127.29.234.1:48363)
0924 16:13:30.859497 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:30.859521 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:30.859678 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:30.860024 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:30.860029 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:30.860517 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:13:30.860686 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:30.860695 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:30.861804 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:30.861812 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:30.862760 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:30.862772 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:30.863513 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:30.863519 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:30.863720 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:30.864390 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:13:30.864408 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:13:30.867863 (+ 3455us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.234.62:37997: Server oryx/127.29.234.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":330,"threads_started":1}
W20260924 16:13:30.868332 7329 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:13:30.858887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.1:48363 (local address 127.29.234.62:37997)
0924 16:13:30.859098 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:30.859106 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:30.859618 (+ 512us) server_negotiation.cc:407] Connection header received
0924 16:13:30.859813 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:30.859819 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:30.859887 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:30.859992 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:30.860849 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:30.861653 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:30.862911 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:30.863322 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:30.863454 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:30.868058 (+ 4604us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.234.1:48363: BlockingRecv error: recv got EOF from 127.29.234.1:48363 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 24 16:13:31 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.3@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:13:31 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.3@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:13:31.109925 7637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:31.100194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:37997 (local address 127.29.234.3:47371)
0924 16:13:31.100734 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:31.100751 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:31.100935 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:31.101290 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:31.101295 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:31.101796 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:13:31.101977 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:31.101986 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:31.103007 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:31.103014 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:31.103943 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:31.103955 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:31.104739 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:31.104746 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:31.104935 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:31.105642 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:13:31.105665 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:13:31.109671 (+ 4006us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.234.62:37997: Server oryx/127.29.234.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":315,"threads_started":1}
W20260924 16:13:31.110109 7329 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:13:31.100294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.3:47371 (local address 127.29.234.62:37997)
0924 16:13:31.100558 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:31.100566 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:31.100851 (+ 285us) server_negotiation.cc:407] Connection header received
0924 16:13:31.101071 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:31.101080 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:31.101145 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:31.101257 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:31.102138 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:31.102868 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:31.104096 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:31.104567 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:31.104704 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:31.109881 (+ 5177us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.234.3:47371: BlockingRecv error: recv got EOF from 127.29.234.3:47371 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
Sep 24 16:13:31 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.2@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:13:31 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.2@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:13:31.475445 7638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:31.465843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:37997 (local address 127.29.234.2:35589)
0924 16:13:31.466501 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:31.466522 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:31.466687 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:31.467122 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:31.467127 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:31.467593 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:13:31.467762 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:31.467772 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:31.468917 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:31.468924 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:31.470050 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:31.470062 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:31.470887 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:31.470894 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:31.471085 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:31.471696 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:13:31.471716 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:13:31.475058 (+ 3342us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.234.62:37997: Server oryx/127.29.234.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":353,"threads_started":1}
W20260924 16:13:31.475533 7329 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:13:31.465872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.2:35589 (local address 127.29.234.62:37997)
0924 16:13:31.466125 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:31.466166 (+ 41us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:31.466614 (+ 448us) server_negotiation.cc:407] Connection header received
0924 16:13:31.466834 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:31.466869 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:31.466954 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:31.467072 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:31.467913 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:31.468761 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:31.470191 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:31.470686 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:31.470844 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:31.475281 (+ 4437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.234.2:35589: BlockingRecv error: recv got EOF from 127.29.234.2:35589 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 24 16:13:31 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.1@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:13:31 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.1@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:13:31.878420 7639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:31.869337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:37997 (local address 127.29.234.1:51857)
0924 16:13:31.869957 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:31.869980 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:31.870123 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:31.870514 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:31.870519 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:31.871049 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:13:31.871206 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:31.871214 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:31.872279 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:31.872286 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:31.873196 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:31.873207 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:31.873991 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:31.873997 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:31.874180 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:31.874765 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:13:31.874782 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:13:31.878192 (+ 3410us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.234.62:37997: Server oryx/127.29.234.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":365,"threads_started":1}
W20260924 16:13:31.878628 7329 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:13:31.869480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.1:51857 (local address 127.29.234.62:37997)
0924 16:13:31.869715 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:31.869722 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:31.870077 (+ 355us) server_negotiation.cc:407] Connection header received
0924 16:13:31.870265 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:31.870272 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:31.870359 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:31.870473 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:31.871366 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:31.872121 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:31.873379 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:31.873831 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:31.873978 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:31.878400 (+ 4422us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.234.1:51857: BlockingRecv error: recv got EOF from 127.29.234.1:51857 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 24 16:13:32 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.3@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:13:32 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.3@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:13:32.120961 7640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:32.110996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:37997 (local address 127.29.234.3:43849)
0924 16:13:32.111579 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:32.111596 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:32.111755 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:32.112099 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:32.112104 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:32.112577 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:13:32.112816 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:32.112825 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:32.113938 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:32.113945 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:32.114883 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:32.114893 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:32.115643 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:32.115650 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:32.115837 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:32.116535 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:13:32.116553 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:13:32.120590 (+ 4037us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.234.62:37997: Server oryx/127.29.234.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":317,"threads_started":1}
W20260924 16:13:32.121021 7329 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:13:32.111093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.3:43849 (local address 127.29.234.62:37997)
0924 16:13:32.111346 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:32.111354 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:32.111681 (+ 327us) server_negotiation.cc:407] Connection header received
0924 16:13:32.111895 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:32.111899 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:32.111960 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:32.112065 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:32.112965 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:32.113791 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:32.115029 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:32.115450 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:32.115583 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:32.120800 (+ 5217us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.234.3:43849: BlockingRecv error: recv got EOF from 127.29.234.3:43849 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 24 16:13:32 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.2@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:13:32 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.2@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:13:32.486451 7641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:32.476600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:37997 (local address 127.29.234.2:57217)
0924 16:13:32.477150 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:32.477174 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:32.477418 (+ 244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:32.477802 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:32.477807 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:32.478280 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:13:32.478438 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:32.478447 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:32.479483 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:32.479490 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:32.480413 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:32.480424 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:32.481411 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:32.481417 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:32.481604 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:32.482288 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:13:32.482306 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:13:32.486053 (+ 3747us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.234.62:37997: Server oryx/127.29.234.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":292,"threads_started":1}
W20260924 16:13:32.486580 7329 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:13:32.476885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.2:57217 (local address 127.29.234.62:37997)
0924 16:13:32.477120 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:32.477127 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:32.477298 (+ 171us) server_negotiation.cc:407] Connection header received
0924 16:13:32.477578 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:32.477589 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:32.477651 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:32.477776 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:32.478598 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:32.479334 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:32.480574 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:32.481211 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:32.481356 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:32.486316 (+ 4960us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.234.2:57217: BlockingRecv error: recv got EOF from 127.29.234.2:57217 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 24 16:13:32 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.1@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:13:32 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.1@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:13:32.888935 7642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:32.879653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:37997 (local address 127.29.234.1:35355)
0924 16:13:32.880165 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:32.880181 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:32.880333 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:32.880689 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:32.880695 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:32.881094 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:13:32.881277 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:32.881286 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:32.882431 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:32.882437 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:32.883496 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:32.883509 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:32.884192 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:32.884196 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:32.884325 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:32.884972 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:13:32.884990 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:13:32.888678 (+ 3688us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.234.62:37997: Server oryx/127.29.234.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":272,"threads_started":1}
W20260924 16:13:32.889061 7329 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:13:32.879743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.1:35355 (local address 127.29.234.62:37997)
0924 16:13:32.879995 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:32.880002 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:32.880264 (+ 262us) server_negotiation.cc:407] Connection header received
0924 16:13:32.880476 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:32.880482 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:32.880549 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:32.880653 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:32.881459 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:32.882282 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:32.883652 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:32.884067 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:32.884164 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:32.888849 (+ 4685us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.234.1:35355: BlockingRecv error: recv got EOF from 127.29.234.1:35355 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 24 16:13:33 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.3@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:13:33 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.3@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:13:33.131980 7643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:33.122014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:37997 (local address 127.29.234.3:44329)
0924 16:13:33.122545 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:33.122561 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:33.122728 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:33.123132 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:33.123137 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:33.123586 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:13:33.123731 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:33.123741 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:33.124833 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:33.124839 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:33.126099 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:33.126123 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:33.126939 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:33.126945 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:33.127133 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:33.127804 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:13:33.127822 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:13:33.131647 (+ 3825us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.234.62:37997: Server oryx/127.29.234.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":294,"threads_started":1}
W20260924 16:13:33.132121 7329 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:13:33.122127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.3:44329 (local address 127.29.234.62:37997)
0924 16:13:33.122456 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:33.122464 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:33.122650 (+ 186us) server_negotiation.cc:407] Connection header received
0924 16:13:33.122922 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:33.122926 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:33.122989 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:33.123098 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:33.123899 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:33.124667 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:33.126285 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:33.126766 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:33.126911 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:33.131867 (+ 4956us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.234.3:44329: BlockingRecv error: recv got EOF from 127.29.234.3:44329 (error 108)
Metrics: {"server-negotiator.queue_time_us":178}
Sep 24 16:13:33 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.2@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 16:13:33 dist-test-slave-v5j9 krb5kdc[6578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.234.2@KRBTEST.COM for oryx/127.29.234.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 16:13:33.497223 7644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:33.487861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:37997 (local address 127.29.234.2:57705)
0924 16:13:33.488468 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:33.488484 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:33.488602 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:33.488984 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:33.488987 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:33.489416 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:13:33.489603 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:33.489612 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:33.490665 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:33.490669 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:33.491586 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:33.491597 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:33.492296 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:33.492302 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:33.492491 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:33.493254 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:13:33.493272 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:13:33.496969 (+ 3697us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.234.62:37997: Server oryx/127.29.234.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":364,"threads_started":1}
W20260924 16:13:33.497502 7329 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:13:33.487955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.2:57705 (local address 127.29.234.62:37997)
0924 16:13:33.488266 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:33.488274 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:33.488578 (+ 304us) server_negotiation.cc:407] Connection header received
0924 16:13:33.488774 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:33.488779 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:33.488847 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:33.488941 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:33.489752 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:33.490538 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:33.491719 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:33.492138 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:33.492264 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:33.497192 (+ 4928us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.234.2:57705: BlockingRecv error: recv got EOF from 127.29.234.2:57705 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
I20260924 16:13:33.869501 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 7176
I20260924 16:13:33.888134 7646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:33.870913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40890 (local address 127.29.234.1:36255)
0924 16:13:33.871685 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:33.871699 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:33.871734 (+ 35us) server_negotiation.cc:407] Connection header received
0924 16:13:33.871804 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:33.871810 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:33.872070 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:33.872326 (+ 256us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:33.877514 (+ 5188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:33.878608 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:33.880431 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:33.881193 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:33.881362 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:33.884204 (+ 2842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:13:33.884214 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:13:33.884231 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:13:33.884282 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:13:33.887316 (+ 3034us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.29.234.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0924 16:13:33.887767 (+ 451us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40890: Request ticket server kudu/127.29.234.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":397,"threads_started":1}
W20260924 16:13:33.888275 7646 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40890: Request ticket server kudu/127.29.234.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260924 16:13:33.888257 30632 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 7176: Not authorized: Client connection negotiation failed: client connection to 127.29.234.1:36255: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.29.234.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260924 16:13:33.888486 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 7176
I20260924 16:13:33.913030 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 7331
W20260924 16:13:33.929879 30632 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 7331: Not authorized: Client connection negotiation failed: client connection to 127.29.234.2:44067: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.29.234.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260924 16:13:33.929992 7647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:33.914362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36416 (local address 127.29.234.2:44067)
0924 16:13:33.915187 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:33.915199 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:33.915239 (+ 40us) server_negotiation.cc:407] Connection header received
0924 16:13:33.915332 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:33.915343 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:33.915688 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:33.915936 (+ 248us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:33.917926 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:33.919821 (+ 1895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:33.921185 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:33.922266 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:33.922426 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:33.925057 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:13:33.925070 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:13:33.925091 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:13:33.925152 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:13:33.929019 (+ 3867us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.29.234.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0924 16:13:33.929594 (+ 575us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36416: Request ticket server kudu/127.29.234.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":402,"threads_started":1}
I20260924 16:13:33.930089 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 7331
W20260924 16:13:33.930085 7647 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36416: Request ticket server kudu/127.29.234.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260924 16:13:33.948889 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 7466
I20260924 16:13:33.962531 7648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:33.950263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49876 (local address 127.29.234.3:38891)
0924 16:13:33.950917 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:33.950927 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:33.950959 (+ 32us) server_negotiation.cc:407] Connection header received
0924 16:13:33.951052 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:33.951062 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:33.951399 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:33.951639 (+ 240us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:33.953009 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:33.954243 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:33.955391 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:33.955960 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:33.956065 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:33.958884 (+ 2819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:13:33.958896 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:13:33.958915 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:13:33.958968 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:13:33.961898 (+ 2930us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.29.234.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0924 16:13:33.962233 (+ 335us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49876: Request ticket server kudu/127.29.234.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":288,"threads_started":1}
W20260924 16:13:33.962652 7648 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49876: Request ticket server kudu/127.29.234.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260924 16:13:33.962736 30632 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 7466: Not authorized: Client connection negotiation failed: client connection to 127.29.234.3:38891: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.29.234.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260924 16:13:33.962949 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 7466
I20260924 16:13:33.981060 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 7105
I20260924 16:13:33.982600 7169 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:13:34.101352 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 7105
2026-09-24T16:13:34Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21658 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-24T16:13:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:13:34Z Disabled control of system clock
I20260924 16:13:34.180802 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266323169505-30632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266323169505-30632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266323169505-30632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:42039
--webserver_interface=127.29.234.62
--webserver_port=0
--builtin_ntp_servers=127.29.234.20:35897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.234.62:42039
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 16:13:34.521037 7653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:13:34.521404 7653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:13:34.521517 7653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:13:34.528846 7653 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:13:34.528970 7653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:13:34.529009 7653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:13:34.529168 7653 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:13:34.529320 7653 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:13:34.539036 7653 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:35897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266323169505-30632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266323169505-30632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.234.62:42039
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:42039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.234.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266323169505-30632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:13:34.540760 7653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:13:34.543010 7653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:13:34.556224 7660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:13:34.556250 7659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:13:34.556180 7662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:13:34.556576 7653 server_base.cc:1034] running on GCE node
I20260924 16:13:34.557708 7653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:13:34.559237 7653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:13:34.560485 7653 hybrid_clock.cc:648] HybridClock initialized: now 1790266414560437 us; error 30 us; skew 500 ppm
I20260924 16:13:34.560899 7653 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 16:13:34.568652 7653 webserver.cc:460] Webserver started at http://127.29.234.62:42909/ using document root <none> and password file <none>
I20260924 16:13:34.569715 7653 fs_manager.cc:359] Metadata directory not provided
I20260924 16:13:34.569831 7653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:13:34.570250 7653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:13:34.573326 7653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266323169505-30632-0/minicluster-data/master-0/data/instance:
uuid: "a84d958defd7441e8fad98e71fc57fe0"
format_stamp: "Formatted at 2026-09-24 16:13:34 on dist-test-slave-v5j9"
server_key: "9bf1ebe93e66418e68d4026e136b50d3"
server_key_iv: "7d5aa973102105926f1171c3e90a3df8"
server_key_version: "encryptionkey@0"
I20260924 16:13:34.574272 7653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266323169505-30632-0/minicluster-data/master-0/wal/instance:
uuid: "a84d958defd7441e8fad98e71fc57fe0"
format_stamp: "Formatted at 2026-09-24 16:13:34 on dist-test-slave-v5j9"
server_key: "9bf1ebe93e66418e68d4026e136b50d3"
server_key_iv: "7d5aa973102105926f1171c3e90a3df8"
server_key_version: "encryptionkey@0"
I20260924 16:13:34.580156 7653 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260924 16:13:34.584146 7668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:13:34.585739 7653 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260924 16:13:34.585930 7653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266323169505-30632-0/minicluster-data/master-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266323169505-30632-0/minicluster-data/master-0/wal
uuid: "a84d958defd7441e8fad98e71fc57fe0"
format_stamp: "Formatted at 2026-09-24 16:13:34 on dist-test-slave-v5j9"
server_key: "9bf1ebe93e66418e68d4026e136b50d3"
server_key_iv: "7d5aa973102105926f1171c3e90a3df8"
server_key_version: "encryptionkey@0"
I20260924 16:13:34.586201 7653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266323169505-30632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266323169505-30632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266323169505-30632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:13:34.605516 7653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:13:34.606463 7653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:13:34.606775 7653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:13:34.628417 7653 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.62:42039
I20260924 16:13:34.628436 7719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.62:42039 every 8 connection(s)
I20260924 16:13:34.630216 7653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
I20260924 16:13:34.634367 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 7653
I20260924 16:13:34.634569 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266323169505-30632-0/minicluster-data/master-0/wal/instance
I20260924 16:13:34.634569 7720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:13:34.635222 30632 external_mini_cluster.cc:1428] Setting key b1dbc1c3144c6ba442fe284439417af9
I20260924 16:13:34.649997 7720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a84d958defd7441e8fad98e71fc57fe0: Bootstrap starting.
I20260924 16:13:34.652185 7723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:34.637437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55672 (local address 127.29.234.62:42039)
0924 16:13:34.638638 (+ 1201us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:34.638659 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:34.638713 (+ 54us) server_negotiation.cc:407] Connection header received
0924 16:13:34.641535 (+ 2822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:34.641567 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:34.642286 (+ 719us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:34.642786 (+ 500us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:34.643332 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:34.645232 (+ 1900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:34.646698 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:34.647849 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:34.648021 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:34.649247 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:13:34.649261 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:13:34.649283 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:13:34.650078 (+ 795us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:13:34.650191 (+ 113us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:13:34.650219 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:13:34.650338 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:34.650531 (+ 193us) server_negotiation.cc:299] Negotiation successful
0924 16:13:34.650876 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":344,"threads_started":1}
I20260924 16:13:34.653929 7720 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a84d958defd7441e8fad98e71fc57fe0: Neither blocks nor log segments found. Creating new log.
I20260924 16:13:34.655272 7720 log.cc:824] T 00000000000000000000000000000000 P a84d958defd7441e8fad98e71fc57fe0: Log is configured to *not* fsync() on all Append() calls
I20260924 16:13:34.658113 7720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a84d958defd7441e8fad98e71fc57fe0: No bootstrap required, opened a new log
I20260924 16:13:34.666093 7720 raft_consensus.cc:348] T 00000000000000000000000000000000 P a84d958defd7441e8fad98e71fc57fe0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a84d958defd7441e8fad98e71fc57fe0" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 42039 } }
I20260924 16:13:34.666703 7720 raft_consensus.cc:374] T 00000000000000000000000000000000 P a84d958defd7441e8fad98e71fc57fe0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:13:34.666841 7720 raft_consensus.cc:729] T 00000000000000000000000000000000 P a84d958defd7441e8fad98e71fc57fe0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a84d958defd7441e8fad98e71fc57fe0, State: Initialized, Role: FOLLOWER
I20260924 16:13:34.667516 7720 consensus_queue.cc:260] T 00000000000000000000000000000000 P a84d958defd7441e8fad98e71fc57fe0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a84d958defd7441e8fad98e71fc57fe0" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 42039 } }
I20260924 16:13:34.667773 7720 raft_consensus.cc:388] T 00000000000000000000000000000000 P a84d958defd7441e8fad98e71fc57fe0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:13:34.667917 7720 raft_consensus.cc:482] T 00000000000000000000000000000000 P a84d958defd7441e8fad98e71fc57fe0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:13:34.668088 7720 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a84d958defd7441e8fad98e71fc57fe0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:13:34.670387 7720 raft_consensus.cc:504] T 00000000000000000000000000000000 P a84d958defd7441e8fad98e71fc57fe0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a84d958defd7441e8fad98e71fc57fe0" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 42039 } }
I20260924 16:13:34.670928 7720 leader_election.cc:302] T 00000000000000000000000000000000 P a84d958defd7441e8fad98e71fc57fe0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a84d958defd7441e8fad98e71fc57fe0; no voters:
I20260924 16:13:34.671530 7720 leader_election.cc:288] T 00000000000000000000000000000000 P a84d958defd7441e8fad98e71fc57fe0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:13:34.671682 7725 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a84d958defd7441e8fad98e71fc57fe0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:13:34.673115 7725 raft_consensus.cc:686] T 00000000000000000000000000000000 P a84d958defd7441e8fad98e71fc57fe0 [term 1 LEADER]: Becoming Leader. State: Replica: a84d958defd7441e8fad98e71fc57fe0, State: Running, Role: LEADER
I20260924 16:13:34.673717 7725 consensus_queue.cc:237] T 00000000000000000000000000000000 P a84d958defd7441e8fad98e71fc57fe0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a84d958defd7441e8fad98e71fc57fe0" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 42039 } }
I20260924 16:13:34.673861 7720 sys_catalog.cc:564] T 00000000000000000000000000000000 P a84d958defd7441e8fad98e71fc57fe0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:13:34.676641 7727 sys_catalog.cc:455] T 00000000000000000000000000000000 P a84d958defd7441e8fad98e71fc57fe0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a84d958defd7441e8fad98e71fc57fe0. Latest consensus state: current_term: 1 leader_uuid: "a84d958defd7441e8fad98e71fc57fe0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a84d958defd7441e8fad98e71fc57fe0" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 42039 } } }
I20260924 16:13:34.677124 7727 sys_catalog.cc:458] T 00000000000000000000000000000000 P a84d958defd7441e8fad98e71fc57fe0 [sys.catalog]: This master's current role is: LEADER
I20260924 16:13:34.676705 7726 sys_catalog.cc:455] T 00000000000000000000000000000000 P a84d958defd7441e8fad98e71fc57fe0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a84d958defd7441e8fad98e71fc57fe0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a84d958defd7441e8fad98e71fc57fe0" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 42039 } } }
I20260924 16:13:34.677371 7726 sys_catalog.cc:458] T 00000000000000000000000000000000 P a84d958defd7441e8fad98e71fc57fe0 [sys.catalog]: This master's current role is: LEADER
I20260924 16:13:34.680806 7734 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:13:34.685223 7734 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:13:34.693255 7734 catalog_manager.cc:1347] Generated new cluster ID: 88246d39377d4612a69576f9b7b8eef7
I20260924 16:13:34.693411 7734 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:13:34.720917 7734 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:13:34.722404 7734 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:13:34.727814 7734 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a84d958defd7441e8fad98e71fc57fe0: Generated new TSK 0
I20260924 16:13:34.728649 7734 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:13:34.741547 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.1:0
--local_ip_for_outbound_sockets=127.29.234.1
--webserver_interface=127.29.234.1
--webserver_port=0
--tserver_master_addrs=127.29.234.62:42039
--builtin_ntp_servers=127.29.234.20:35897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260924 16:13:35.077164 7744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:13:35.077593 7744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:13:35.077759 7744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:13:35.085425 7744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:13:35.085568 7744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:13:35.085757 7744 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.1
I20260924 16:13:35.095554 7744 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:35897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.234.1
--webserver_port=0
--tserver_master_addrs=127.29.234.62:42039
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.1
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:13:35.096958 7744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:13:35.098888 7744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:13:35.112571 7749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:13:35.112612 7750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:13:35.112809 7753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:13:35.113626 7744 server_base.cc:1034] running on GCE node
I20260924 16:13:35.114281 7744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:13:35.115242 7744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:13:35.116456 7744 hybrid_clock.cc:648] HybridClock initialized: now 1790266415116385 us; error 66 us; skew 500 ppm
I20260924 16:13:35.116787 7744 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 16:13:35.119125 7744 webserver.cc:460] Webserver started at http://127.29.234.1:45593/ using document root <none> and password file <none>
I20260924 16:13:35.119846 7744 fs_manager.cc:359] Metadata directory not provided
I20260924 16:13:35.119968 7744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:13:35.120329 7744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:13:35.123441 7744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-0/data/instance:
uuid: "725d3e14e3fb4c7abeb33cfc4f46db97"
format_stamp: "Formatted at 2026-09-24 16:13:35 on dist-test-slave-v5j9"
server_key: "aa32be39fd4f79ab12abb28ef814752d"
server_key_iv: "ed84296a87fdf912c1930ad3041e49e2"
server_key_version: "encryptionkey@0"
I20260924 16:13:35.124248 7744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-0/wal/instance:
uuid: "725d3e14e3fb4c7abeb33cfc4f46db97"
format_stamp: "Formatted at 2026-09-24 16:13:35 on dist-test-slave-v5j9"
server_key: "aa32be39fd4f79ab12abb28ef814752d"
server_key_iv: "ed84296a87fdf912c1930ad3041e49e2"
server_key_version: "encryptionkey@0"
I20260924 16:13:35.129693 7744 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260924 16:13:35.133245 7759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:13:35.134737 7744 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:13:35.134920 7744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-0/wal
uuid: "725d3e14e3fb4c7abeb33cfc4f46db97"
format_stamp: "Formatted at 2026-09-24 16:13:35 on dist-test-slave-v5j9"
server_key: "aa32be39fd4f79ab12abb28ef814752d"
server_key_iv: "ed84296a87fdf912c1930ad3041e49e2"
server_key_version: "encryptionkey@0"
I20260924 16:13:35.135221 7744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:13:35.155664 7744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:13:35.156636 7744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:13:35.156981 7744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:13:35.158074 7744 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:13:35.159575 7744 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:13:35.159693 7744 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:13:35.159824 7744 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:13:35.159930 7744 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:13:35.194188 7744 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.1:38019
I20260924 16:13:35.194248 7873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.1:38019 every 8 connection(s)
I20260924 16:13:35.195619 7744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
I20260924 16:13:35.204850 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 7744
I20260924 16:13:35.205060 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-0/wal/instance
I20260924 16:13:35.205785 30632 external_mini_cluster.cc:1428] Setting key 80189413d7655381388198a4d23e5f07
I20260924 16:13:35.208958 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.2:0
--local_ip_for_outbound_sockets=127.29.234.2
--webserver_interface=127.29.234.2
--webserver_port=0
--tserver_master_addrs=127.29.234.62:42039
--builtin_ntp_servers=127.29.234.20:35897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 16:13:35.210196 7876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:35.198490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.1:58131 (local address 127.29.234.62:42039)
0924 16:13:35.198964 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:35.198968 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:35.200258 (+ 1290us) server_negotiation.cc:407] Connection header received
0924 16:13:35.202321 (+ 2063us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:35.202326 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:35.202434 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:35.202565 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:35.203790 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:35.204560 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:35.206645 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:35.207161 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:35.207274 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:35.209093 (+ 1819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:13:35.209102 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:13:35.209110 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:13:35.209138 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:13:35.209195 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:13:35.209217 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:13:35.209323 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:35.209614 (+ 291us) server_negotiation.cc:299] Negotiation successful
0924 16:13:35.209741 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":370,"thread_start_us":258,"threads_started":1}
I20260924 16:13:35.211207 7877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:35.199050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:42039 (local address 127.29.234.1:58131)
0924 16:13:35.200063 (+ 1013us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:35.200109 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:35.201987 (+ 1878us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:35.202848 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:35.202858 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:35.202959 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:13:35.203590 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:35.203606 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:35.204753 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:35.204757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:35.206421 (+ 1664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:35.206439 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:35.207431 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:35.207439 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:35.207825 (+ 386us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:35.208698 (+ 873us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:13:35.208701 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:13:35.208750 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:13:35.208851 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:13:35.209325 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:13:35.209334 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:13:35.209380 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 16:13:35.209567 (+ 187us) client_negotiation.cc:240] Negotiation successful
0924 16:13:35.209953 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":325,"threads_started":1}
I20260924 16:13:35.213001 7874 heartbeater.cc:347] Connected to a master server at 127.29.234.62:42039
I20260924 16:13:35.213399 7874 heartbeater.cc:464] Registering TS with master...
I20260924 16:13:35.214287 7874 heartbeater.cc:511] Master 127.29.234.62:42039 requested a full tablet report, sending...
I20260924 16:13:35.216517 7685 ts_manager.cc:194] Registered new tserver with Master: 725d3e14e3fb4c7abeb33cfc4f46db97 (127.29.234.1:38019)
I20260924 16:13:35.218199 7685 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.234.1:58131
W20260924 16:13:35.547190 7878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:13:35.547561 7878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:13:35.547734 7878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:13:35.555259 7878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:13:35.555424 7878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:13:35.555609 7878 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.2
I20260924 16:13:35.565207 7878 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:35897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.234.2
--webserver_port=0
--tserver_master_addrs=127.29.234.62:42039
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.2
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:13:35.566733 7878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:13:35.568729 7878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:13:35.582226 7883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:13:35.582302 7886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:13:35.582252 7884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:13:35.583530 7878 server_base.cc:1034] running on GCE node
I20260924 16:13:35.584162 7878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:13:35.585170 7878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:13:35.586444 7878 hybrid_clock.cc:648] HybridClock initialized: now 1790266415586382 us; error 51 us; skew 500 ppm
I20260924 16:13:35.586786 7878 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 16:13:35.589151 7878 webserver.cc:460] Webserver started at http://127.29.234.2:40889/ using document root <none> and password file <none>
I20260924 16:13:35.589969 7878 fs_manager.cc:359] Metadata directory not provided
I20260924 16:13:35.590111 7878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:13:35.590492 7878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:13:35.593698 7878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-1/data/instance:
uuid: "737654c1b86846269dc9747525c7f44b"
format_stamp: "Formatted at 2026-09-24 16:13:35 on dist-test-slave-v5j9"
server_key: "1578cbe46a3326cf8bd7a099bff02120"
server_key_iv: "d3e4d23f19192e565b5dec4477c7fbb6"
server_key_version: "encryptionkey@0"
I20260924 16:13:35.594535 7878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-1/wal/instance:
uuid: "737654c1b86846269dc9747525c7f44b"
format_stamp: "Formatted at 2026-09-24 16:13:35 on dist-test-slave-v5j9"
server_key: "1578cbe46a3326cf8bd7a099bff02120"
server_key_iv: "d3e4d23f19192e565b5dec4477c7fbb6"
server_key_version: "encryptionkey@0"
I20260924 16:13:35.599931 7878 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 16:13:35.603724 7893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:13:35.605283 7878 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 16:13:35.605509 7878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-1/wal
uuid: "737654c1b86846269dc9747525c7f44b"
format_stamp: "Formatted at 2026-09-24 16:13:35 on dist-test-slave-v5j9"
server_key: "1578cbe46a3326cf8bd7a099bff02120"
server_key_iv: "d3e4d23f19192e565b5dec4477c7fbb6"
server_key_version: "encryptionkey@0"
I20260924 16:13:35.605767 7878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:13:35.621435 7878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:13:35.622386 7878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:13:35.622720 7878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:13:35.623694 7878 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:13:35.625200 7878 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:13:35.625331 7878 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:13:35.625507 7878 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:13:35.625597 7878 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:13:35.661588 7878 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.2:45413
I20260924 16:13:35.661634 8008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.2:45413 every 8 connection(s)
I20260924 16:13:35.663003 7878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
I20260924 16:13:35.663766 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 7878
I20260924 16:13:35.663985 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-1/wal/instance
I20260924 16:13:35.664654 30632 external_mini_cluster.cc:1428] Setting key 3f52e1ce40190ce5a1fd8ab395da0b0a
I20260924 16:13:35.668048 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.3:0
--local_ip_for_outbound_sockets=127.29.234.3
--webserver_interface=127.29.234.3
--webserver_port=0
--tserver_master_addrs=127.29.234.62:42039
--builtin_ntp_servers=127.29.234.20:35897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 16:13:35.677094 7876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:35.666172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.2:34535 (local address 127.29.234.62:42039)
0924 16:13:35.666398 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:35.666406 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:35.668079 (+ 1673us) server_negotiation.cc:407] Connection header received
0924 16:13:35.670131 (+ 2052us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:35.670137 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:35.670239 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:35.670364 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:35.671548 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:35.672280 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:35.674033 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:35.674538 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:35.674670 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:35.676237 (+ 1567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:13:35.676242 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:13:35.676246 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:13:35.676268 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:13:35.676300 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:13:35.676319 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:13:35.676413 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:35.676669 (+ 256us) server_negotiation.cc:299] Negotiation successful
0924 16:13:35.676772 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 16:13:35.678232 8011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:35.666916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:42039 (local address 127.29.234.2:34535)
0924 16:13:35.667928 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:35.667964 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:35.669800 (+ 1836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:35.670630 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:35.670639 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:35.670742 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:13:35.671373 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:35.671391 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:35.672460 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:35.672464 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:35.673886 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:35.673898 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:35.674779 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:35.674787 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:35.675054 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:35.675879 (+ 825us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:13:35.675882 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:13:35.675925 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:13:35.676020 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:13:35.676414 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:13:35.676428 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:13:35.676439 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 16:13:35.676642 (+ 203us) client_negotiation.cc:240] Negotiation successful
0924 16:13:35.677086 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":332,"threads_started":1}
I20260924 16:13:35.679800 8009 heartbeater.cc:347] Connected to a master server at 127.29.234.62:42039
I20260924 16:13:35.680131 8009 heartbeater.cc:464] Registering TS with master...
I20260924 16:13:35.680940 8009 heartbeater.cc:511] Master 127.29.234.62:42039 requested a full tablet report, sending...
I20260924 16:13:35.682523 7685 ts_manager.cc:194] Registered new tserver with Master: 737654c1b86846269dc9747525c7f44b (127.29.234.2:45413)
I20260924 16:13:35.683794 7685 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.234.2:34535
W20260924 16:13:36.016120 8012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:13:36.016647 8012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:13:36.016745 8012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:13:36.024149 8012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:13:36.024277 8012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:13:36.024384 8012 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.3
I20260924 16:13:36.033661 8012 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:35897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.234.3
--webserver_port=0
--tserver_master_addrs=127.29.234.62:42039
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.3
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:13:36.034924 8012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:13:36.036707 8012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:13:36.050829 8018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:13:36.050784 8020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:13:36.050881 8017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:13:36.055133 8012 server_base.cc:1034] running on GCE node
I20260924 16:13:36.055936 8012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:13:36.057114 8012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:13:36.058383 8012 hybrid_clock.cc:648] HybridClock initialized: now 1790266416058313 us; error 80 us; skew 500 ppm
I20260924 16:13:36.058812 8012 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 16:13:36.061414 8012 webserver.cc:460] Webserver started at http://127.29.234.3:43263/ using document root <none> and password file <none>
I20260924 16:13:36.062244 8012 fs_manager.cc:359] Metadata directory not provided
I20260924 16:13:36.062374 8012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:13:36.062743 8012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:13:36.065871 8012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-2/data/instance:
uuid: "10a1c2ff2a9b47f881f18c21bf23e9d5"
format_stamp: "Formatted at 2026-09-24 16:13:36 on dist-test-slave-v5j9"
server_key: "eee85084596207e811d14e64fe1e4959"
server_key_iv: "6637c5c70a57ac15fd9dccfefa0027cd"
server_key_version: "encryptionkey@0"
I20260924 16:13:36.066747 8012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-2/wal/instance:
uuid: "10a1c2ff2a9b47f881f18c21bf23e9d5"
format_stamp: "Formatted at 2026-09-24 16:13:36 on dist-test-slave-v5j9"
server_key: "eee85084596207e811d14e64fe1e4959"
server_key_iv: "6637c5c70a57ac15fd9dccfefa0027cd"
server_key_version: "encryptionkey@0"
I20260924 16:13:36.072394 8012 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260924 16:13:36.076109 8026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:13:36.077754 8012 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 16:13:36.077939 8012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-2/wal
uuid: "10a1c2ff2a9b47f881f18c21bf23e9d5"
format_stamp: "Formatted at 2026-09-24 16:13:36 on dist-test-slave-v5j9"
server_key: "eee85084596207e811d14e64fe1e4959"
server_key_iv: "6637c5c70a57ac15fd9dccfefa0027cd"
server_key_version: "encryptionkey@0"
I20260924 16:13:36.078163 8012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:13:36.093236 8012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:13:36.094087 8012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:13:36.094393 8012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:13:36.095357 8012 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:13:36.096767 8012 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:13:36.096887 8012 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:13:36.097026 8012 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:13:36.097115 8012 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:13:36.131063 8012 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.3:39821
I20260924 16:13:36.131115 8140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.3:39821 every 8 connection(s)
I20260924 16:13:36.132396 8012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
I20260924 16:13:36.133064 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 8012
I20260924 16:13:36.133239 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-2/wal/instance
I20260924 16:13:36.133911 30632 external_mini_cluster.cc:1428] Setting key c4c27aae73482dc23bfb644ed4346373
I20260924 16:13:36.145447 7876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:36.135291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.3:46945 (local address 127.29.234.62:42039)
0924 16:13:36.135511 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:36.135517 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:36.137321 (+ 1804us) server_negotiation.cc:407] Connection header received
0924 16:13:36.139325 (+ 2004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:36.139329 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:36.139397 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:36.139474 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:36.140619 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:36.141433 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:36.142942 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:36.143362 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:36.143451 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:36.144717 (+ 1266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:13:36.144721 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:13:36.144724 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:13:36.144742 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:13:36.144776 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:13:36.144788 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:13:36.144857 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:36.145042 (+ 185us) server_negotiation.cc:299] Negotiation successful
0924 16:13:36.145118 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 16:13:36.146409 8143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:36.136085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:42039 (local address 127.29.234.3:46945)
0924 16:13:36.137149 (+ 1064us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:36.137208 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:36.139016 (+ 1808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:36.139672 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:36.139679 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:36.139762 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:13:36.140472 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:36.140491 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:36.141558 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:36.141561 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:36.142811 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:36.142822 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:36.143536 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:36.143540 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:36.143744 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:36.144429 (+ 685us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:13:36.144432 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:13:36.144469 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:13:36.144539 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:13:36.144870 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:13:36.144878 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:13:36.144883 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 16:13:36.145016 (+ 133us) client_negotiation.cc:240] Negotiation successful
0924 16:13:36.145289 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":342,"threads_started":1}
I20260924 16:13:36.147693 8141 heartbeater.cc:347] Connected to a master server at 127.29.234.62:42039
I20260924 16:13:36.148047 8141 heartbeater.cc:464] Registering TS with master...
I20260924 16:13:36.149087 8141 heartbeater.cc:511] Master 127.29.234.62:42039 requested a full tablet report, sending...
I20260924 16:13:36.150661 7685 ts_manager.cc:194] Registered new tserver with Master: 10a1c2ff2a9b47f881f18c21bf23e9d5 (127.29.234.3:39821)
I20260924 16:13:36.151489 7685 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.234.3:46945
I20260924 16:13:36.162374 30632 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 16:13:36.168869 7876 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:13:36.167256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55682 (local address 127.29.234.62:42039)
0924 16:13:36.167500 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:36.167505 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:36.167961 (+ 456us) server_negotiation.cc:407] Connection header received
0924 16:13:36.168104 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:36.168108 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:36.168198 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:36.168295 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:36.168646 (+ 351us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55682: BlockingRecv error: recv got EOF from 127.0.0.1:55682 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 16:13:36.169549 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 7744
I20260924 16:13:36.178283 8151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:36.170839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38198 (local address 127.29.234.1:38019)
0924 16:13:36.171444 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:36.171452 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:36.171481 (+ 29us) server_negotiation.cc:407] Connection header received
0924 16:13:36.171554 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:36.171561 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:36.171814 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:36.171952 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:36.172284 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:36.173451 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:36.174962 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:36.175561 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:36.175678 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:36.176752 (+ 1074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:13:36.176761 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:13:36.176777 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:13:36.177509 (+ 732us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:13:36.177564 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:13:36.177587 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:13:36.177687 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:36.177862 (+ 175us) server_negotiation.cc:299] Negotiation successful
0924 16:13:36.177993 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":328,"threads_started":1}
I20260924 16:13:36.179580 7869 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:13:36.261912 7874 heartbeater.cc:503] Master 127.29.234.62:42039 was elected leader, sending a full tablet report...
I20260924 16:13:36.268255 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 7744
I20260924 16:13:36.286027 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 7878
I20260924 16:13:36.294445 8153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:36.287003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53054 (local address 127.29.234.2:45413)
0924 16:13:36.287498 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:36.287505 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:36.287531 (+ 26us) server_negotiation.cc:407] Connection header received
0924 16:13:36.287598 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:36.287604 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:36.287819 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:36.287949 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:36.288355 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:36.289434 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:36.290784 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:36.291487 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:36.291595 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:36.292741 (+ 1146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:13:36.292749 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:13:36.292772 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:13:36.293537 (+ 765us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:13:36.293587 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:13:36.293614 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:13:36.293707 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:36.293897 (+ 190us) server_negotiation.cc:299] Negotiation successful
0924 16:13:36.294059 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":241,"threads_started":1}
I20260924 16:13:36.295845 8004 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:13:36.377760 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 7878
I20260924 16:13:36.395236 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 8012
I20260924 16:13:36.403891 8155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:36.396217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54892 (local address 127.29.234.3:39821)
0924 16:13:36.396924 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:36.396931 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:36.396964 (+ 33us) server_negotiation.cc:407] Connection header received
0924 16:13:36.397032 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:36.397039 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:36.397325 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:36.397574 (+ 249us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:36.398131 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:36.399217 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:36.400519 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:36.401102 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:36.401259 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:36.402382 (+ 1123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:13:36.402391 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:13:36.402407 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:13:36.403022 (+ 615us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:13:36.403076 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:13:36.403097 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:13:36.403237 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:36.403387 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 16:13:36.403540 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":386,"threads_started":1}
I20260924 16:13:36.405197 8136 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:13:36.488142 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 8012
I20260924 16:13:36.504739 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 7653
I20260924 16:13:36.506033 7715 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:13:36.612560 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 7653
2026-09-24T16:13:36Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2506 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-24T16:13:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:13:36Z Disabled control of system clock
I20260924 16:13:36.687633 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266323169505-30632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266323169505-30632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266323169505-30632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:33399
--webserver_interface=127.29.234.62
--webserver_port=0
--builtin_ntp_servers=127.29.234.20:37405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.234.62:33399
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 16:13:37.027505 8161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:13:37.027814 8161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:13:37.027901 8161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:13:37.035316 8161 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:13:37.035447 8161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:13:37.035485 8161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:13:37.035535 8161 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:13:37.035580 8161 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:13:37.045125 8161 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:37405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266323169505-30632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266323169505-30632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.234.62:33399
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:33399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.234.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266323169505-30632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:13:37.046639 8161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:13:37.048666 8161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:13:37.061512 8166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:13:37.062057 8167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:13:37.062108 8169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:13:37.062301 8161 server_base.cc:1034] running on GCE node
I20260924 16:13:37.063344 8161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:13:37.064949 8161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:13:37.066219 8161 hybrid_clock.cc:648] HybridClock initialized: now 1790266417066151 us; error 103 us; skew 500 ppm
I20260924 16:13:37.066651 8161 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 16:13:37.069273 8161 webserver.cc:460] Webserver started at http://127.29.234.62:38283/ using document root <none> and password file <none>
I20260924 16:13:37.070103 8161 fs_manager.cc:359] Metadata directory not provided
I20260924 16:13:37.070227 8161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:13:37.070587 8161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:13:37.075069 8161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266323169505-30632-0/minicluster-data/master-0/data/instance:
uuid: "d8b39d74233645259efd879dfda7f91f"
format_stamp: "Formatted at 2026-09-24 16:13:37 on dist-test-slave-v5j9"
server_key: "2314c16927b45ca81c32b6c37e9f9150"
server_key_iv: "68493872dcdf04a6956d1657d4aa8518"
server_key_version: "encryptionkey@0"
I20260924 16:13:37.075878 8161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266323169505-30632-0/minicluster-data/master-0/wal/instance:
uuid: "d8b39d74233645259efd879dfda7f91f"
format_stamp: "Formatted at 2026-09-24 16:13:37 on dist-test-slave-v5j9"
server_key: "2314c16927b45ca81c32b6c37e9f9150"
server_key_iv: "68493872dcdf04a6956d1657d4aa8518"
server_key_version: "encryptionkey@0"
I20260924 16:13:37.081115 8161 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 16:13:37.084709 8175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:13:37.086318 8161 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:13:37.086504 8161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266323169505-30632-0/minicluster-data/master-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266323169505-30632-0/minicluster-data/master-0/wal
uuid: "d8b39d74233645259efd879dfda7f91f"
format_stamp: "Formatted at 2026-09-24 16:13:37 on dist-test-slave-v5j9"
server_key: "2314c16927b45ca81c32b6c37e9f9150"
server_key_iv: "68493872dcdf04a6956d1657d4aa8518"
server_key_version: "encryptionkey@0"
I20260924 16:13:37.086743 8161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266323169505-30632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266323169505-30632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266323169505-30632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:13:37.092774 8161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:13:37.093654 8161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:13:37.093963 8161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:13:37.112447 8161 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.62:33399
I20260924 16:13:37.112505 8226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.62:33399 every 8 connection(s)
I20260924 16:13:37.113926 8161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
I20260924 16:13:37.118080 8227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:13:37.120222 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 8161
I20260924 16:13:37.120455 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266323169505-30632-0/minicluster-data/master-0/wal/instance
I20260924 16:13:37.121069 30632 external_mini_cluster.cc:1428] Setting key 093eeb430d9e768236189ce954b5bb7a
I20260924 16:13:37.129743 8230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:37.123223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34916 (local address 127.29.234.62:33399)
0924 16:13:37.124313 (+ 1090us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:37.124324 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:37.124367 (+ 43us) server_negotiation.cc:407] Connection header received
0924 16:13:37.126105 (+ 1738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:37.126144 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:37.126564 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:37.126902 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:37.128099 (+ 1197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:13:37.128105 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:13:37.128118 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:13:37.128135 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:13:37.128193 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:13:37.128208 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:13:37.128395 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:37.128491 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 16:13:37.128768 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":674,"thread_start_us":411,"threads_started":1}
I20260924 16:13:37.130505 8227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8b39d74233645259efd879dfda7f91f: Bootstrap starting.
I20260924 16:13:37.133591 8227 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8b39d74233645259efd879dfda7f91f: Neither blocks nor log segments found. Creating new log.
I20260924 16:13:37.134634 8227 log.cc:824] T 00000000000000000000000000000000 P d8b39d74233645259efd879dfda7f91f: Log is configured to *not* fsync() on all Append() calls
I20260924 16:13:37.137699 8227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8b39d74233645259efd879dfda7f91f: No bootstrap required, opened a new log
I20260924 16:13:37.145268 8227 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8b39d74233645259efd879dfda7f91f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8b39d74233645259efd879dfda7f91f" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 33399 } }
I20260924 16:13:37.145763 8227 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8b39d74233645259efd879dfda7f91f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:13:37.145876 8227 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8b39d74233645259efd879dfda7f91f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8b39d74233645259efd879dfda7f91f, State: Initialized, Role: FOLLOWER
I20260924 16:13:37.146572 8227 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8b39d74233645259efd879dfda7f91f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8b39d74233645259efd879dfda7f91f" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 33399 } }
I20260924 16:13:37.146862 8227 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8b39d74233645259efd879dfda7f91f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:13:37.147034 8227 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8b39d74233645259efd879dfda7f91f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:13:37.147254 8227 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8b39d74233645259efd879dfda7f91f [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:13:37.149533 8227 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8b39d74233645259efd879dfda7f91f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8b39d74233645259efd879dfda7f91f" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 33399 } }
I20260924 16:13:37.150084 8227 leader_election.cc:302] T 00000000000000000000000000000000 P d8b39d74233645259efd879dfda7f91f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8b39d74233645259efd879dfda7f91f; no voters:
I20260924 16:13:37.150707 8227 leader_election.cc:288] T 00000000000000000000000000000000 P d8b39d74233645259efd879dfda7f91f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:13:37.150846 8232 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8b39d74233645259efd879dfda7f91f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:13:37.151621 8232 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8b39d74233645259efd879dfda7f91f [term 1 LEADER]: Becoming Leader. State: Replica: d8b39d74233645259efd879dfda7f91f, State: Running, Role: LEADER
I20260924 16:13:37.152243 8232 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8b39d74233645259efd879dfda7f91f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8b39d74233645259efd879dfda7f91f" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 33399 } }
I20260924 16:13:37.153043 8227 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8b39d74233645259efd879dfda7f91f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:13:37.155329 8234 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8b39d74233645259efd879dfda7f91f [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8b39d74233645259efd879dfda7f91f. Latest consensus state: current_term: 1 leader_uuid: "d8b39d74233645259efd879dfda7f91f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8b39d74233645259efd879dfda7f91f" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 33399 } } }
I20260924 16:13:37.155459 8233 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8b39d74233645259efd879dfda7f91f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8b39d74233645259efd879dfda7f91f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8b39d74233645259efd879dfda7f91f" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 33399 } } }
I20260924 16:13:37.155809 8233 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8b39d74233645259efd879dfda7f91f [sys.catalog]: This master's current role is: LEADER
I20260924 16:13:37.155807 8234 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8b39d74233645259efd879dfda7f91f [sys.catalog]: This master's current role is: LEADER
I20260924 16:13:37.162868 8240 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:13:37.167433 8240 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:13:37.176549 8240 catalog_manager.cc:1347] Generated new cluster ID: 155ad621fe474eabb6f7ceabe0e91944
I20260924 16:13:37.176743 8240 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:13:37.212257 8240 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:13:37.212752 8240 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:13:37.220506 8240 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8b39d74233645259efd879dfda7f91f: Generated new TSK 0
I20260924 16:13:37.221665 8240 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:13:37.228102 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.1:0
--local_ip_for_outbound_sockets=127.29.234.1
--webserver_interface=127.29.234.1
--webserver_port=0
--tserver_master_addrs=127.29.234.62:33399
--builtin_ntp_servers=127.29.234.20:37405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 16:13:37.572080 8251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:13:37.572451 8251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:13:37.572597 8251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:13:37.579968 8251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:13:37.580091 8251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:13:37.580209 8251 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.1
I20260924 16:13:37.589860 8251 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:37405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.234.1
--webserver_port=0
--tserver_master_addrs=127.29.234.62:33399
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.1
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:13:37.591321 8251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:13:37.593223 8251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:13:37.606901 8257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:13:37.607086 8258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:13:37.606801 8260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:13:37.608631 8251 server_base.cc:1034] running on GCE node
I20260924 16:13:37.609539 8251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:13:37.610630 8251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:13:37.611912 8251 hybrid_clock.cc:648] HybridClock initialized: now 1790266417611856 us; error 63 us; skew 500 ppm
I20260924 16:13:37.612325 8251 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 16:13:37.614996 8251 webserver.cc:460] Webserver started at http://127.29.234.1:36309/ using document root <none> and password file <none>
I20260924 16:13:37.615806 8251 fs_manager.cc:359] Metadata directory not provided
I20260924 16:13:37.615950 8251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:13:37.616308 8251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:13:37.619626 8251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-0/data/instance:
uuid: "8d78f4250a7440b2b9ec8399fd5e52c1"
format_stamp: "Formatted at 2026-09-24 16:13:37 on dist-test-slave-v5j9"
server_key: "1e0378c19f96ade87b67e78d710aec7c"
server_key_iv: "f18f0fdfa4f2dc52cbc5f66415b79423"
server_key_version: "encryptionkey@0"
I20260924 16:13:37.620532 8251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-0/wal/instance:
uuid: "8d78f4250a7440b2b9ec8399fd5e52c1"
format_stamp: "Formatted at 2026-09-24 16:13:37 on dist-test-slave-v5j9"
server_key: "1e0378c19f96ade87b67e78d710aec7c"
server_key_iv: "f18f0fdfa4f2dc52cbc5f66415b79423"
server_key_version: "encryptionkey@0"
I20260924 16:13:37.627502 8251 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260924 16:13:37.631644 8266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:13:37.633049 8251 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 16:13:37.633261 8251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-0/wal
uuid: "8d78f4250a7440b2b9ec8399fd5e52c1"
format_stamp: "Formatted at 2026-09-24 16:13:37 on dist-test-slave-v5j9"
server_key: "1e0378c19f96ade87b67e78d710aec7c"
server_key_iv: "f18f0fdfa4f2dc52cbc5f66415b79423"
server_key_version: "encryptionkey@0"
I20260924 16:13:37.633563 8251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:13:37.639358 8251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:13:37.640225 8251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:13:37.640511 8251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:13:37.641578 8251 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:13:37.642985 8251 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:13:37.643105 8251 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:13:37.643236 8251 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:13:37.643322 8251 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:13:37.676760 8251 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.1:40679
I20260924 16:13:37.676806 8379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.1:40679 every 8 connection(s)
I20260924 16:13:37.678288 8251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
I20260924 16:13:37.684862 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 8251
I20260924 16:13:37.685096 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-0/wal/instance
I20260924 16:13:37.685904 30632 external_mini_cluster.cc:1428] Setting key 342952ebb5bc87c2514dcda75b20c656
I20260924 16:13:37.687295 8382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:37.681006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.1:42681 (local address 127.29.234.62:33399)
0924 16:13:37.681492 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:37.681499 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:37.682839 (+ 1340us) server_negotiation.cc:407] Connection header received
0924 16:13:37.684624 (+ 1785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:37.684629 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:37.684702 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:37.684835 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:37.686384 (+ 1549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:13:37.686390 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:13:37.686395 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:13:37.686414 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:13:37.686463 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:13:37.686479 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:13:37.686607 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:37.686873 (+ 266us) server_negotiation.cc:299] Negotiation successful
0924 16:13:37.686942 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":223,"threads_started":1}
I20260924 16:13:37.688304 8383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:37.681687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:33399 (local address 127.29.234.1:42681)
0924 16:13:37.682631 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:37.682668 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:37.684343 (+ 1675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:37.685095 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:37.685105 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:37.685244 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:13:37.686066 (+ 822us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:13:37.686069 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:13:37.686108 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:13:37.686207 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:13:37.686597 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:13:37.686608 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:13:37.686616 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 16:13:37.686920 (+ 304us) client_negotiation.cc:240] Negotiation successful
0924 16:13:37.687187 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":275,"threads_started":1}
I20260924 16:13:37.688941 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.2:0
--local_ip_for_outbound_sockets=127.29.234.2
--webserver_interface=127.29.234.2
--webserver_port=0
--tserver_master_addrs=127.29.234.62:33399
--builtin_ntp_servers=127.29.234.20:37405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260924 16:13:37.689953 8380 heartbeater.cc:347] Connected to a master server at 127.29.234.62:33399
I20260924 16:13:37.690302 8380 heartbeater.cc:464] Registering TS with master...
I20260924 16:13:37.690961 8380 heartbeater.cc:511] Master 127.29.234.62:33399 requested a full tablet report, sending...
I20260924 16:13:37.693253 8192 ts_manager.cc:194] Registered new tserver with Master: 8d78f4250a7440b2b9ec8399fd5e52c1 (127.29.234.1:40679)
W20260924 16:13:38.020043 8384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:13:38.020336 8384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:13:38.020432 8384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:13:38.027771 8384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:13:38.027889 8384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:13:38.027997 8384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.2
I20260924 16:13:38.037413 8384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:37405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.234.2
--webserver_port=0
--tserver_master_addrs=127.29.234.62:33399
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.2
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:13:38.038836 8384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:13:38.040660 8384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:13:38.054245 8389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:13:38.054210 8392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:13:38.054145 8390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:13:38.055132 8384 server_base.cc:1034] running on GCE node
I20260924 16:13:38.055806 8384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:13:38.056842 8384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:13:38.058075 8384 hybrid_clock.cc:648] HybridClock initialized: now 1790266418058012 us; error 56 us; skew 500 ppm
I20260924 16:13:38.058424 8384 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 16:13:38.060889 8384 webserver.cc:460] Webserver started at http://127.29.234.2:46463/ using document root <none> and password file <none>
I20260924 16:13:38.061743 8384 fs_manager.cc:359] Metadata directory not provided
I20260924 16:13:38.061861 8384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:13:38.062206 8384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:13:38.065232 8384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-1/data/instance:
uuid: "4022f8b30b7b43a99b01935927c58575"
format_stamp: "Formatted at 2026-09-24 16:13:38 on dist-test-slave-v5j9"
server_key: "6abb5eebfde5c846ec34eedff2a2177c"
server_key_iv: "c22ad061c9a9d63d7f91992dad96da78"
server_key_version: "encryptionkey@0"
I20260924 16:13:38.066123 8384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-1/wal/instance:
uuid: "4022f8b30b7b43a99b01935927c58575"
format_stamp: "Formatted at 2026-09-24 16:13:38 on dist-test-slave-v5j9"
server_key: "6abb5eebfde5c846ec34eedff2a2177c"
server_key_iv: "c22ad061c9a9d63d7f91992dad96da78"
server_key_version: "encryptionkey@0"
I20260924 16:13:38.071674 8384 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 16:13:38.075402 8399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:13:38.076756 8384 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 16:13:38.076926 8384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-1/wal
uuid: "4022f8b30b7b43a99b01935927c58575"
format_stamp: "Formatted at 2026-09-24 16:13:38 on dist-test-slave-v5j9"
server_key: "6abb5eebfde5c846ec34eedff2a2177c"
server_key_iv: "c22ad061c9a9d63d7f91992dad96da78"
server_key_version: "encryptionkey@0"
I20260924 16:13:38.077157 8384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:13:38.082971 8384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:13:38.083804 8384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:13:38.084088 8384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:13:38.085044 8384 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:13:38.086555 8384 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:13:38.086640 8384 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:13:38.086800 8384 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:13:38.086890 8384 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:13:38.120838 8384 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.2:40423
I20260924 16:13:38.120885 8512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.2:40423 every 8 connection(s)
I20260924 16:13:38.122259 8384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
I20260924 16:13:38.123852 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 8384
I20260924 16:13:38.124037 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-1/wal/instance
I20260924 16:13:38.124559 30632 external_mini_cluster.cc:1428] Setting key 409174c1d7cfe26cc61ec4f5d8883d56
I20260924 16:13:38.127955 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.3:0
--local_ip_for_outbound_sockets=127.29.234.3
--webserver_interface=127.29.234.3
--webserver_port=0
--tserver_master_addrs=127.29.234.62:33399
--builtin_ntp_servers=127.29.234.20:37405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260924 16:13:38.133770 8382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:38.127087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.2:36855 (local address 127.29.234.62:33399)
0924 16:13:38.127271 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:38.127277 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:38.128825 (+ 1548us) server_negotiation.cc:407] Connection header received
0924 16:13:38.131330 (+ 2505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:38.131338 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:38.131410 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:38.131516 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:38.132938 (+ 1422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:13:38.132947 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:13:38.132954 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:13:38.132978 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:13:38.133024 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:13:38.133043 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:13:38.133172 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:38.133338 (+ 166us) server_negotiation.cc:299] Negotiation successful
0924 16:13:38.133426 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 16:13:38.134946 8515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:38.127510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:33399 (local address 127.29.234.2:36855)
0924 16:13:38.128662 (+ 1152us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:38.128708 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:38.130995 (+ 2287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:38.131766 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:38.131776 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:38.131877 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:13:38.132591 (+ 714us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:13:38.132594 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:13:38.132635 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:13:38.132728 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:13:38.133142 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:13:38.133150 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:13:38.133157 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 16:13:38.133464 (+ 307us) client_negotiation.cc:240] Negotiation successful
0924 16:13:38.133691 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":341,"threads_started":1}
I20260924 16:13:38.136526 8513 heartbeater.cc:347] Connected to a master server at 127.29.234.62:33399
I20260924 16:13:38.136861 8513 heartbeater.cc:464] Registering TS with master...
I20260924 16:13:38.137676 8513 heartbeater.cc:511] Master 127.29.234.62:33399 requested a full tablet report, sending...
I20260924 16:13:38.139252 8192 ts_manager.cc:194] Registered new tserver with Master: 4022f8b30b7b43a99b01935927c58575 (127.29.234.2:40423)
W20260924 16:13:38.466611 8516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:13:38.466885 8516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:13:38.466974 8516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:13:38.474073 8516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:13:38.474180 8516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:13:38.474282 8516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.3
I20260924 16:13:38.483240 8516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:37405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.234.3
--webserver_port=0
--tserver_master_addrs=127.29.234.62:33399
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.3
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:13:38.484521 8516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:13:38.486361 8516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:13:38.500048 8521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:13:38.499980 8524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:13:38.500038 8522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:13:38.500950 8516 server_base.cc:1034] running on GCE node
I20260924 16:13:38.501675 8516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:13:38.502668 8516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:13:38.503904 8516 hybrid_clock.cc:648] HybridClock initialized: now 1790266418503865 us; error 47 us; skew 500 ppm
I20260924 16:13:38.504248 8516 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 16:13:38.506558 8516 webserver.cc:460] Webserver started at http://127.29.234.3:33799/ using document root <none> and password file <none>
I20260924 16:13:38.507296 8516 fs_manager.cc:359] Metadata directory not provided
I20260924 16:13:38.507428 8516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:13:38.507762 8516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:13:38.510754 8516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-2/data/instance:
uuid: "2637a0e2ee184f1f8e311a1e72ec53a3"
format_stamp: "Formatted at 2026-09-24 16:13:38 on dist-test-slave-v5j9"
server_key: "a29af64798cbb7d20795adf44c905613"
server_key_iv: "2af5e46d56fe50284e4365a3c6338e9d"
server_key_version: "encryptionkey@0"
I20260924 16:13:38.511556 8516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-2/wal/instance:
uuid: "2637a0e2ee184f1f8e311a1e72ec53a3"
format_stamp: "Formatted at 2026-09-24 16:13:38 on dist-test-slave-v5j9"
server_key: "a29af64798cbb7d20795adf44c905613"
server_key_iv: "2af5e46d56fe50284e4365a3c6338e9d"
server_key_version: "encryptionkey@0"
I20260924 16:13:38.516708 8516 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260924 16:13:38.520221 8530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:13:38.521687 8516 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:13:38.521874 8516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-2/wal
uuid: "2637a0e2ee184f1f8e311a1e72ec53a3"
format_stamp: "Formatted at 2026-09-24 16:13:38 on dist-test-slave-v5j9"
server_key: "a29af64798cbb7d20795adf44c905613"
server_key_iv: "2af5e46d56fe50284e4365a3c6338e9d"
server_key_version: "encryptionkey@0"
I20260924 16:13:38.522113 8516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:13:38.527947 8516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:13:38.528774 8516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:13:38.529055 8516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:13:38.530019 8516 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:13:38.531464 8516 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:13:38.531589 8516 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:13:38.531726 8516 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:13:38.531818 8516 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:13:38.564646 8516 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.3:42471
I20260924 16:13:38.564699 8643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.3:42471 every 8 connection(s)
I20260924 16:13:38.566008 8516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
I20260924 16:13:38.573660 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 8516
I20260924 16:13:38.573792 8382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:38.568537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.3:53847 (local address 127.29.234.62:33399)
0924 16:13:38.568720 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:38.568724 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:38.570091 (+ 1367us) server_negotiation.cc:407] Connection header received
0924 16:13:38.571705 (+ 1614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:38.571709 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:38.571758 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:38.571840 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:38.573060 (+ 1220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:13:38.573065 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:13:38.573068 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:13:38.573084 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:13:38.573115 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:13:38.573126 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:13:38.573233 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:38.573495 (+ 262us) server_negotiation.cc:299] Negotiation successful
0924 16:13:38.573548 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 16:13:38.573884 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790266323169505-30632-0/minicluster-data/ts-2/wal/instance
I20260924 16:13:38.574433 30632 external_mini_cluster.cc:1428] Setting key 88b0dc6db2e19df82dbf87de66ba7c39
I20260924 16:13:38.575102 8646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:38.569068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:33399 (local address 127.29.234.3:53847)
0924 16:13:38.569964 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:38.570006 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:38.571470 (+ 1464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:38.572039 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:38.572059 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:38.572148 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:13:38.572782 (+ 634us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:13:38.572785 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:13:38.572823 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:13:38.572906 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:13:38.573237 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:13:38.573249 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:13:38.573265 (+ 16us) client_negotiation.cc:769] Sending connection context
0924 16:13:38.573593 (+ 328us) client_negotiation.cc:240] Negotiation successful
0924 16:13:38.573877 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":276,"threads_started":1}
I20260924 16:13:38.576598 8644 heartbeater.cc:347] Connected to a master server at 127.29.234.62:33399
I20260924 16:13:38.576893 8644 heartbeater.cc:464] Registering TS with master...
I20260924 16:13:38.577579 8644 heartbeater.cc:511] Master 127.29.234.62:33399 requested a full tablet report, sending...
I20260924 16:13:38.578891 8192 ts_manager.cc:194] Registered new tserver with Master: 2637a0e2ee184f1f8e311a1e72ec53a3 (127.29.234.3:42471)
I20260924 16:13:38.589994 30632 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 16:13:38.596064 8382 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:13:38.594368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34928 (local address 127.29.234.62:33399)
0924 16:13:38.594546 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:38.594551 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:38.595070 (+ 519us) server_negotiation.cc:407] Connection header received
0924 16:13:38.595254 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:38.595260 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:38.595324 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:38.595427 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:38.595777 (+ 350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34928: BlockingRecv error: recv got EOF from 127.0.0.1:34928 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 16:13:38.597020 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 8251
I20260924 16:13:38.600659 8653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:38.598001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60500 (local address 127.29.234.1:40679)
0924 16:13:38.598531 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:38.598538 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:38.598565 (+ 27us) server_negotiation.cc:407] Connection header received
0924 16:13:38.598633 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:38.598639 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:38.598889 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:38.599014 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:38.599887 (+ 873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:13:38.599900 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:13:38.599920 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:13:38.599942 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:13:38.600014 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:13:38.600038 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:13:38.600182 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:38.600264 (+ 82us) server_negotiation.cc:299] Negotiation successful
0924 16:13:38.600399 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":254,"threads_started":1}
I20260924 16:13:38.602073 8375 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:13:38.685761 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 8251
I20260924 16:13:38.703804 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 8384
I20260924 16:13:38.707496 8655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:38.704794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54500 (local address 127.29.234.2:40423)
0924 16:13:38.705307 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:38.705314 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:38.705391 (+ 77us) server_negotiation.cc:407] Connection header received
0924 16:13:38.705473 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:38.705480 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:38.705725 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:38.705854 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:38.706779 (+ 925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:13:38.706786 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:13:38.706800 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:13:38.706816 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:13:38.706855 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:13:38.706869 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:13:38.706969 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:38.707082 (+ 113us) server_negotiation.cc:299] Negotiation successful
0924 16:13:38.707201 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":248,"threads_started":1}
I20260924 16:13:38.708722 8508 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:13:38.790052 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 8384
I20260924 16:13:38.807605 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 8516
I20260924 16:13:38.811829 8657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:38.808743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38534 (local address 127.29.234.3:42471)
0924 16:13:38.809478 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:38.809486 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:38.809520 (+ 34us) server_negotiation.cc:407] Connection header received
0924 16:13:38.809589 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:38.809596 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:38.809835 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:38.809978 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:38.811075 (+ 1097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:13:38.811081 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:13:38.811094 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:13:38.811111 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:13:38.811152 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:13:38.811166 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:13:38.811300 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:38.811355 (+ 55us) server_negotiation.cc:299] Negotiation successful
0924 16:13:38.811477 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":350,"threads_started":1}
I20260924 16:13:38.813165 8639 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:13:38.895898 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 8516
I20260924 16:13:38.912786 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 8161
I20260924 16:13:38.914047 8222 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:13:39.019960 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 8161
2026-09-24T16:13:39Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2407 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:13:39 dist-test-slave-v5j9 krb5kdc[8662](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:13:39 dist-test-slave-v5j9 krb5kdc[8662](info): set up 2 sockets
Sep 24 16:13:39 dist-test-slave-v5j9 krb5kdc[8662](info): commencing operation
krb5kdc: starting...
W20260924 16:13:40.133616 30632 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.006s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Sep 24 16:13:40 dist-test-slave-v5j9 krb5kdc[8662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266420, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:13:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:13:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.234.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab.
Entry for principal kudu/127.29.234.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab.
I20260924 16:13:40.372994 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:42321
--webserver_interface=127.29.234.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab
--principal=kudu/127.29.234.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:46297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.234.62:42321
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:13:40.710261 8676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:13:40.710716 8676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:13:40.711074 8676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:13:40.723858 8676 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:13:40.724030 8676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:13:40.724115 8676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:13:40.724201 8676 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:13:40.724275 8676 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:13:40.738837 8676 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:46297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.234.62:42321
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab
--principal=kudu/127.29.234.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:42321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.234.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:13:40.740389 8676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:13:40.742097 8676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:13:40.761685 8681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:13:40.761814 8684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:13:40.762580 8676 server_base.cc:1034] running on GCE node
W20260924 16:13:40.762669 8682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:13:40.763500 8676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:13:40.764999 8676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:13:40.766242 8676 hybrid_clock.cc:648] HybridClock initialized: now 1790266420766192 us; error 30 us; skew 500 ppm
I20260924 16:13:40.766628 8676 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:13:40 dist-test-slave-v5j9 krb5kdc[8662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266420, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:13:40.769734 8676 init.cc:377] Logged in from keytab as kudu/127.29.234.62@KRBTEST.COM (short username kudu)
I20260924 16:13:40.771215 8676 webserver.cc:460] Webserver started at http://127.29.234.62:35039/ using document root <none> and password file <none>
I20260924 16:13:40.771966 8676 fs_manager.cc:359] Metadata directory not provided
I20260924 16:13:40.772119 8676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:13:40.772481 8676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:13:40.775724 8676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/minicluster-data/master-0/data/instance:
uuid: "36d0d7e738254dfc906c44c86c75166e"
format_stamp: "Formatted at 2026-09-24 16:13:40 on dist-test-slave-v5j9"
server_key: "3bfba7a0ae5c730edbd5d2ea9c520a77"
server_key_iv: "d757b21e03aca75aab69ab20d8a4f47e"
server_key_version: "encryptionkey@0"
I20260924 16:13:40.776520 8676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/minicluster-data/master-0/wal/instance:
uuid: "36d0d7e738254dfc906c44c86c75166e"
format_stamp: "Formatted at 2026-09-24 16:13:40 on dist-test-slave-v5j9"
server_key: "3bfba7a0ae5c730edbd5d2ea9c520a77"
server_key_iv: "d757b21e03aca75aab69ab20d8a4f47e"
server_key_version: "encryptionkey@0"
I20260924 16:13:40.781949 8676 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260924 16:13:40.785318 8691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:13:40.786892 8676 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:13:40.787037 8676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/minicluster-data/master-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/minicluster-data/master-0/wal
uuid: "36d0d7e738254dfc906c44c86c75166e"
format_stamp: "Formatted at 2026-09-24 16:13:40 on dist-test-slave-v5j9"
server_key: "3bfba7a0ae5c730edbd5d2ea9c520a77"
server_key_iv: "d757b21e03aca75aab69ab20d8a4f47e"
server_key_version: "encryptionkey@0"
I20260924 16:13:40.787204 8676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:13:40.803664 8676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:13:40.808027 8676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:13:40.808279 8676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:13:40.825659 8676 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.62:42321
I20260924 16:13:40.825695 8743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.62:42321 every 8 connection(s)
I20260924 16:13:40.827205 8676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
I20260924 16:13:40.831107 8744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:13:40.836943 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 8676
I20260924 16:13:40.837190 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/minicluster-data/master-0/wal/instance
I20260924 16:13:40.837872 30632 external_mini_cluster.cc:1428] Setting key 11d18d8a84765924f1fff8c0b678205d
I20260924 16:13:40.843194 8744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36d0d7e738254dfc906c44c86c75166e: Bootstrap starting.
I20260924 16:13:40.847154 8744 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36d0d7e738254dfc906c44c86c75166e: Neither blocks nor log segments found. Creating new log.
I20260924 16:13:40.848393 8744 log.cc:824] T 00000000000000000000000000000000 P 36d0d7e738254dfc906c44c86c75166e: Log is configured to *not* fsync() on all Append() calls
I20260924 16:13:40.851790 8744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36d0d7e738254dfc906c44c86c75166e: No bootstrap required, opened a new log
Sep 24 16:13:40 dist-test-slave-v5j9 krb5kdc[8662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266420, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:13:40.862001 8744 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36d0d7e738254dfc906c44c86c75166e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36d0d7e738254dfc906c44c86c75166e" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 42321 } }
I20260924 16:13:40.862461 8744 raft_consensus.cc:374] T 00000000000000000000000000000000 P 36d0d7e738254dfc906c44c86c75166e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:13:40.862569 8744 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36d0d7e738254dfc906c44c86c75166e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36d0d7e738254dfc906c44c86c75166e, State: Initialized, Role: FOLLOWER
I20260924 16:13:40.863246 8744 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36d0d7e738254dfc906c44c86c75166e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36d0d7e738254dfc906c44c86c75166e" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 42321 } }
I20260924 16:13:40.863526 8744 raft_consensus.cc:388] T 00000000000000000000000000000000 P 36d0d7e738254dfc906c44c86c75166e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:13:40.863669 8744 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36d0d7e738254dfc906c44c86c75166e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:13:40.863881 8747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:40.840036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40098 (local address 127.29.234.62:42321)
0924 16:13:40.840874 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:40.840891 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:40.840942 (+ 51us) server_negotiation.cc:407] Connection header received
0924 16:13:40.843323 (+ 2381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:40.843357 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:40.843870 (+ 513us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:40.844314 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:40.845699 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:40.847922 (+ 2223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:40.849573 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:40.850566 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:40.850762 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:40.854685 (+ 3923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:13:40.854695 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:13:40.854721 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:13:40.854765 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:13:40.858914 (+ 4149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:40.859720 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:40.859735 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:40.859744 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:40.859981 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:40.860391 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:40.860401 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:40.860405 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:40.861186 (+ 781us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:13:40.861535 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:40.861943 (+ 408us) server_negotiation.cc:299] Negotiation successful
0924 16:13:40.862442 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":355,"threads_started":1}
I20260924 16:13:40.863891 8744 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36d0d7e738254dfc906c44c86c75166e [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:13:40.866951 8744 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36d0d7e738254dfc906c44c86c75166e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36d0d7e738254dfc906c44c86c75166e" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 42321 } }
I20260924 16:13:40.867573 8744 leader_election.cc:302] T 00000000000000000000000000000000 P 36d0d7e738254dfc906c44c86c75166e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36d0d7e738254dfc906c44c86c75166e; no voters:
I20260924 16:13:40.868099 8744 leader_election.cc:288] T 00000000000000000000000000000000 P 36d0d7e738254dfc906c44c86c75166e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:13:40.868242 8749 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36d0d7e738254dfc906c44c86c75166e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:13:40.868911 8749 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36d0d7e738254dfc906c44c86c75166e [term 1 LEADER]: Becoming Leader. State: Replica: 36d0d7e738254dfc906c44c86c75166e, State: Running, Role: LEADER
I20260924 16:13:40.869530 8749 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36d0d7e738254dfc906c44c86c75166e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36d0d7e738254dfc906c44c86c75166e" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 42321 } }
I20260924 16:13:40.871563 8744 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36d0d7e738254dfc906c44c86c75166e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:13:40.872464 8750 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36d0d7e738254dfc906c44c86c75166e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36d0d7e738254dfc906c44c86c75166e. Latest consensus state: current_term: 1 leader_uuid: "36d0d7e738254dfc906c44c86c75166e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36d0d7e738254dfc906c44c86c75166e" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 42321 } } }
I20260924 16:13:40.872429 8751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36d0d7e738254dfc906c44c86c75166e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36d0d7e738254dfc906c44c86c75166e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36d0d7e738254dfc906c44c86c75166e" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 42321 } } }
I20260924 16:13:40.872857 8750 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36d0d7e738254dfc906c44c86c75166e [sys.catalog]: This master's current role is: LEADER
I20260924 16:13:40.872862 8751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36d0d7e738254dfc906c44c86c75166e [sys.catalog]: This master's current role is: LEADER
I20260924 16:13:40.874624 8754 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:13:40.879320 8754 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:13:40.889101 8754 catalog_manager.cc:1347] Generated new cluster ID: a3f021ce93934b1e83bc2eb83c12db07
I20260924 16:13:40.889216 8754 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:13:40.909193 8754 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:13:40.911149 8754 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:13:40.931658 8754 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 36d0d7e738254dfc906c44c86c75166e: Generated new TSK 0
I20260924 16:13:40.932916 8754 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.234.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab.
Entry for principal kudu/127.29.234.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab.
I20260924 16:13:40.996807 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.1:0
--local_ip_for_outbound_sockets=127.29.234.1
--webserver_interface=127.29.234.1
--webserver_port=0
--tserver_master_addrs=127.29.234.62:42321
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab
--principal=kudu/127.29.234.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:46297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:13:41.329388 8770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:13:41.329679 8770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:13:41.329850 8770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:13:41.336913 8770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:13:41.337015 8770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:13:41.337124 8770 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.1
I20260924 16:13:41.346791 8770 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:46297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab
--principal=kudu/127.29.234.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.234.1
--webserver_port=0
--tserver_master_addrs=127.29.234.62:42321
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.1
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:13:41.348181 8770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:13:41.349699 8770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:13:41.363250 8779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:13:41.363284 8777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:13:41.363811 8776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:13:41.364784 8770 server_base.cc:1034] running on GCE node
I20260924 16:13:41.365418 8770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:13:41.366439 8770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:13:41.367633 8770 hybrid_clock.cc:648] HybridClock initialized: now 1790266421367586 us; error 49 us; skew 500 ppm
I20260924 16:13:41.367970 8770 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:13:41 dist-test-slave-v5j9 krb5kdc[8662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266421, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:13:41.371384 8770 init.cc:377] Logged in from keytab as kudu/127.29.234.1@KRBTEST.COM (short username kudu)
I20260924 16:13:41.373085 8770 webserver.cc:460] Webserver started at http://127.29.234.1:46645/ using document root <none> and password file <none>
I20260924 16:13:41.374006 8770 fs_manager.cc:359] Metadata directory not provided
I20260924 16:13:41.374131 8770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:13:41.374462 8770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:13:41.377664 8770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/minicluster-data/ts-0/data/instance:
uuid: "a913e5c7c99e40a4ba83897d03cb1108"
format_stamp: "Formatted at 2026-09-24 16:13:41 on dist-test-slave-v5j9"
server_key: "1585312f0a735c7d7e07db77cffbe2c8"
server_key_iv: "d5024e8e4cee14b0cbbe6b1e578aaaab"
server_key_version: "encryptionkey@0"
I20260924 16:13:41.378470 8770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/minicluster-data/ts-0/wal/instance:
uuid: "a913e5c7c99e40a4ba83897d03cb1108"
format_stamp: "Formatted at 2026-09-24 16:13:41 on dist-test-slave-v5j9"
server_key: "1585312f0a735c7d7e07db77cffbe2c8"
server_key_iv: "d5024e8e4cee14b0cbbe6b1e578aaaab"
server_key_version: "encryptionkey@0"
I20260924 16:13:41.384037 8770 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 16:13:41.387828 8786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:13:41.389324 8770 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 16:13:41.389545 8770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/minicluster-data/ts-0/wal
uuid: "a913e5c7c99e40a4ba83897d03cb1108"
format_stamp: "Formatted at 2026-09-24 16:13:41 on dist-test-slave-v5j9"
server_key: "1585312f0a735c7d7e07db77cffbe2c8"
server_key_iv: "d5024e8e4cee14b0cbbe6b1e578aaaab"
server_key_version: "encryptionkey@0"
I20260924 16:13:41.389751 8770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:13:41.412199 8770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:13:41.416263 8770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:13:41.416548 8770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:13:41.417572 8770 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:13:41.418931 8770 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:13:41.419005 8770 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:13:41.419159 8770 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:13:41.419251 8770 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:13:41.452628 8770 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.1:42169
I20260924 16:13:41.452678 8901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.1:42169 every 8 connection(s)
I20260924 16:13:41.453975 8770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
I20260924 16:13:41.460475 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 8770
I20260924 16:13:41.460731 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/minicluster-data/ts-0/wal/instance
I20260924 16:13:41.461367 30632 external_mini_cluster.cc:1428] Setting key 3faf1b0520597657542df15de5d1c8e2
Sep 24 16:13:41 dist-test-slave-v5j9 krb5kdc[8662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266421, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.1@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:13:41.476119 8904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:41.456716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.1:34993 (local address 127.29.234.62:42321)
0924 16:13:41.457236 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:41.457241 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:41.458156 (+ 915us) server_negotiation.cc:407] Connection header received
0924 16:13:41.460020 (+ 1864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:41.460026 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:41.460123 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:41.460252 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:41.462015 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:41.462919 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:41.464629 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:41.465432 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:41.465585 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:41.469491 (+ 3906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:13:41.469497 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:13:41.469500 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:13:41.469532 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:13:41.472674 (+ 3142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:41.473556 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:41.473564 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:41.473571 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:41.473718 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:41.474120 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:41.474128 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:41.474134 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:41.474508 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:13:41.474658 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:41.475364 (+ 706us) server_negotiation.cc:299] Negotiation successful
0924 16:13:41.475620 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":285,"threads_started":1}
I20260924 16:13:41.476840 8905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:41.457242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:42321 (local address 127.29.234.1:34993)
0924 16:13:41.457993 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:41.458037 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:41.459726 (+ 1689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:41.460501 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:41.460511 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:41.461123 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:13:41.461836 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:41.461859 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:41.463130 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:41.463135 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:41.464458 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:41.464470 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:41.465645 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:41.465650 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:41.465845 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:41.466561 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:13:41.466585 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:13:41.469217 (+ 2632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:13:41.472813 (+ 3596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:13:41.472820 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:13:41.472834 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:13:41.473430 (+ 596us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:13:41.473822 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:13:41.473825 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:13:41.473828 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:13:41.474007 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:13:41.474628 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:13:41.474635 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:13:41.474964 (+ 329us) client_negotiation.cc:769] Sending connection context
0924 16:13:41.475199 (+ 235us) client_negotiation.cc:240] Negotiation successful
0924 16:13:41.475626 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":298,"threads_started":1}
I20260924 16:13:41.478564 8902 heartbeater.cc:347] Connected to a master server at 127.29.234.62:42321
I20260924 16:13:41.478905 8902 heartbeater.cc:464] Registering TS with master...
I20260924 16:13:41.479681 8902 heartbeater.cc:511] Master 127.29.234.62:42321 requested a full tablet report, sending...
I20260924 16:13:41.482030 8709 ts_manager.cc:194] Registered new tserver with Master: a913e5c7c99e40a4ba83897d03cb1108 (127.29.234.1:42169)
I20260924 16:13:41.483755 8709 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.234.1@KRBTEST.COM'} at 127.29.234.1:34993
WARNING: no policy specified for kudu/127.29.234.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab.
Entry for principal kudu/127.29.234.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab.
I20260924 16:13:41.517745 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.2:0
--local_ip_for_outbound_sockets=127.29.234.2
--webserver_interface=127.29.234.2
--webserver_port=0
--tserver_master_addrs=127.29.234.62:42321
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab
--principal=kudu/127.29.234.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:46297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:13:41.850953 8908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:13:41.851269 8908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:13:41.851450 8908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:13:41.859313 8908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:13:41.859472 8908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:13:41.859647 8908 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.2
I20260924 16:13:41.869691 8908 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:46297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab
--principal=kudu/127.29.234.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.234.2
--webserver_port=0
--tserver_master_addrs=127.29.234.62:42321
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.2
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:13:41.871057 8908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:13:41.872483 8908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:13:41.885771 8916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:13:41.885702 8913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:13:41.885743 8914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:13:41.886895 8908 server_base.cc:1034] running on GCE node
I20260924 16:13:41.887485 8908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:13:41.888417 8908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:13:41.889607 8908 hybrid_clock.cc:648] HybridClock initialized: now 1790266421889548 us; error 54 us; skew 500 ppm
I20260924 16:13:41.889933 8908 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:13:41 dist-test-slave-v5j9 krb5kdc[8662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266421, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:13:41.893221 8908 init.cc:377] Logged in from keytab as kudu/127.29.234.2@KRBTEST.COM (short username kudu)
I20260924 16:13:41.894812 8908 webserver.cc:460] Webserver started at http://127.29.234.2:43167/ using document root <none> and password file <none>
I20260924 16:13:41.895550 8908 fs_manager.cc:359] Metadata directory not provided
I20260924 16:13:41.895692 8908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:13:41.896003 8908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:13:41.899080 8908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/minicluster-data/ts-1/data/instance:
uuid: "c005f1f00903408199b4debd9d2e1916"
format_stamp: "Formatted at 2026-09-24 16:13:41 on dist-test-slave-v5j9"
server_key: "2a14bc757be6306208228103660b2fcb"
server_key_iv: "2ff61fec8107620d5761d2e76a7b803b"
server_key_version: "encryptionkey@0"
I20260924 16:13:41.899854 8908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/minicluster-data/ts-1/wal/instance:
uuid: "c005f1f00903408199b4debd9d2e1916"
format_stamp: "Formatted at 2026-09-24 16:13:41 on dist-test-slave-v5j9"
server_key: "2a14bc757be6306208228103660b2fcb"
server_key_iv: "2ff61fec8107620d5761d2e76a7b803b"
server_key_version: "encryptionkey@0"
I20260924 16:13:41.905069 8908 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 16:13:41.908514 8923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:13:41.909947 8908 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:13:41.910125 8908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/minicluster-data/ts-1/wal
uuid: "c005f1f00903408199b4debd9d2e1916"
format_stamp: "Formatted at 2026-09-24 16:13:41 on dist-test-slave-v5j9"
server_key: "2a14bc757be6306208228103660b2fcb"
server_key_iv: "2ff61fec8107620d5761d2e76a7b803b"
server_key_version: "encryptionkey@0"
I20260924 16:13:41.910303 8908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:13:41.935336 8908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:13:41.939549 8908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:13:41.939847 8908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:13:41.940819 8908 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:13:41.942226 8908 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:13:41.942299 8908 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:13:41.942461 8908 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:13:41.942513 8908 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:13:41.975634 8908 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.2:40583
I20260924 16:13:41.975677 9038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.2:40583 every 8 connection(s)
I20260924 16:13:41.977025 8908 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
I20260924 16:13:41.980098 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 8908
I20260924 16:13:41.980327 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/minicluster-data/ts-1/wal/instance
I20260924 16:13:41.980870 30632 external_mini_cluster.cc:1428] Setting key 003e965f51cc1a482208ab294c2105e1
Sep 24 16:13:41 dist-test-slave-v5j9 krb5kdc[8662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266421, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.2@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:13:41.999265 9041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:41.979888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.2:56973 (local address 127.29.234.62:42321)
0924 16:13:41.980391 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:41.980399 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:41.981745 (+ 1346us) server_negotiation.cc:407] Connection header received
0924 16:13:41.983746 (+ 2001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:41.983752 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:41.983841 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:41.983952 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:41.985463 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:41.986203 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:41.987742 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:41.988152 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:41.988291 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:41.992141 (+ 3850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:13:41.992150 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:13:41.992158 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:13:41.992204 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:13:41.995860 (+ 3656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:41.996618 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:41.996626 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:41.996632 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:41.996794 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:41.997208 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:41.997216 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:41.997222 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:41.997649 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:13:41.997802 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:41.998502 (+ 700us) server_negotiation.cc:299] Negotiation successful
0924 16:13:41.998758 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":247,"threads_started":1}
I20260924 16:13:41.999877 9042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:41.980645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:42321 (local address 127.29.234.2:56973)
0924 16:13:41.981547 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:41.981600 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:41.983495 (+ 1895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:41.984180 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:41.984189 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:41.984731 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:13:41.985279 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:41.985295 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:41.986366 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:41.986371 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:41.987602 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:41.987613 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:41.988367 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:41.988374 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:41.988596 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:41.989295 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:13:41.989318 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:13:41.991890 (+ 2572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:13:41.996007 (+ 4117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:13:41.996015 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:13:41.996030 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:13:41.996476 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:13:41.996899 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:13:41.996902 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:13:41.996904 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:13:41.997079 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:13:41.997778 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:13:41.997783 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:13:41.998108 (+ 325us) client_negotiation.cc:769] Sending connection context
0924 16:13:41.998337 (+ 229us) client_negotiation.cc:240] Negotiation successful
0924 16:13:41.998746 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":343,"threads_started":1}
I20260924 16:13:42.001452 9039 heartbeater.cc:347] Connected to a master server at 127.29.234.62:42321
I20260924 16:13:42.001768 9039 heartbeater.cc:464] Registering TS with master...
I20260924 16:13:42.002532 9039 heartbeater.cc:511] Master 127.29.234.62:42321 requested a full tablet report, sending...
I20260924 16:13:42.004066 8709 ts_manager.cc:194] Registered new tserver with Master: c005f1f00903408199b4debd9d2e1916 (127.29.234.2:40583)
I20260924 16:13:42.004874 8709 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.234.2@KRBTEST.COM'} at 127.29.234.2:56973
WARNING: no policy specified for kudu/127.29.234.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab.
Entry for principal kudu/127.29.234.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab.
I20260924 16:13:42.036212 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.3:0
--local_ip_for_outbound_sockets=127.29.234.3
--webserver_interface=127.29.234.3
--webserver_port=0
--tserver_master_addrs=127.29.234.62:42321
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab
--principal=kudu/127.29.234.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:46297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:13:42.380259 9045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:13:42.380610 9045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:13:42.380790 9045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:13:42.388574 9045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:13:42.388741 9045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:13:42.388922 9045 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.3
I20260924 16:13:42.399577 9045 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:46297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab
--principal=kudu/127.29.234.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.234.3
--webserver_port=0
--tserver_master_addrs=127.29.234.62:42321
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.3
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:13:42.401049 9045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:13:42.402514 9045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:13:42.416288 9050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:13:42.416253 9051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:13:42.416409 9053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:13:42.417294 9045 server_base.cc:1034] running on GCE node
I20260924 16:13:42.418015 9045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:13:42.419040 9045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:13:42.420296 9045 hybrid_clock.cc:648] HybridClock initialized: now 1790266422420217 us; error 64 us; skew 500 ppm
I20260924 16:13:42.420656 9045 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:13:42 dist-test-slave-v5j9 krb5kdc[8662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266422, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:13:42.423964 9045 init.cc:377] Logged in from keytab as kudu/127.29.234.3@KRBTEST.COM (short username kudu)
I20260924 16:13:42.425657 9045 webserver.cc:460] Webserver started at http://127.29.234.3:36893/ using document root <none> and password file <none>
I20260924 16:13:42.426450 9045 fs_manager.cc:359] Metadata directory not provided
I20260924 16:13:42.426604 9045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:13:42.426925 9045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:13:42.430066 9045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/minicluster-data/ts-2/data/instance:
uuid: "06090350da8c44b1a413c98119175766"
format_stamp: "Formatted at 2026-09-24 16:13:42 on dist-test-slave-v5j9"
server_key: "71d5df42a505371a00322ba932b241bc"
server_key_iv: "f091eadae23960f88dea15a6bfc440f7"
server_key_version: "encryptionkey@0"
I20260924 16:13:42.430845 9045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/minicluster-data/ts-2/wal/instance:
uuid: "06090350da8c44b1a413c98119175766"
format_stamp: "Formatted at 2026-09-24 16:13:42 on dist-test-slave-v5j9"
server_key: "71d5df42a505371a00322ba932b241bc"
server_key_iv: "f091eadae23960f88dea15a6bfc440f7"
server_key_version: "encryptionkey@0"
I20260924 16:13:42.435964 9045 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260924 16:13:42.439606 9060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:13:42.440938 9045 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 16:13:42.441119 9045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/minicluster-data/ts-2/wal
uuid: "06090350da8c44b1a413c98119175766"
format_stamp: "Formatted at 2026-09-24 16:13:42 on dist-test-slave-v5j9"
server_key: "71d5df42a505371a00322ba932b241bc"
server_key_iv: "f091eadae23960f88dea15a6bfc440f7"
server_key_version: "encryptionkey@0"
I20260924 16:13:42.441318 9045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:13:42.452848 9045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:13:42.456925 9045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:13:42.457218 9045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:13:42.458225 9045 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:13:42.459492 9045 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:13:42.459599 9045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:13:42.459748 9045 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:13:42.459822 9045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:13:42.487011 8902 heartbeater.cc:503] Master 127.29.234.62:42321 was elected leader, sending a full tablet report...
I20260924 16:13:42.494685 9045 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.3:44941
I20260924 16:13:42.494727 9174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.3:44941 every 8 connection(s)
I20260924 16:13:42.496068 9045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
I20260924 16:13:42.498980 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 9045
I20260924 16:13:42.499200 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/minicluster-data/ts-2/wal/instance
I20260924 16:13:42.499634 30632 external_mini_cluster.cc:1428] Setting key 5bfff5688f2f1d302a18018318986b96
Sep 24 16:13:42 dist-test-slave-v5j9 krb5kdc[8662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266422, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.3@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:13:42.516474 9041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:42.498738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.3:43963 (local address 127.29.234.62:42321)
0924 16:13:42.498969 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:42.498977 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:42.500334 (+ 1357us) server_negotiation.cc:407] Connection header received
0924 16:13:42.502622 (+ 2288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:42.502627 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:42.502733 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:42.502845 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:42.504367 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:42.505094 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:42.506720 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:42.507113 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:42.507278 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:42.510650 (+ 3372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:13:42.510655 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:13:42.510658 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:13:42.510695 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:13:42.513225 (+ 2530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:42.514151 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:42.514158 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:42.514162 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:42.514329 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:42.514849 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:42.514854 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:42.514857 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:42.515255 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:13:42.515375 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:42.516047 (+ 672us) server_negotiation.cc:299] Negotiation successful
0924 16:13:42.516245 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 16:13:42.517513 9177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:42.499334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:42321 (local address 127.29.234.3:43963)
0924 16:13:42.500169 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:42.500223 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:42.502342 (+ 2119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:42.503082 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:42.503091 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:42.503596 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:13:42.504186 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:42.504206 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:42.505260 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:42.505266 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:42.506563 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:42.506579 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:42.507368 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:42.507387 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:42.507641 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:42.508252 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:13:42.508274 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:13:42.510453 (+ 2179us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:13:42.513414 (+ 2961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:13:42.513421 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:13:42.513434 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:13:42.513992 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:13:42.514452 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:13:42.514459 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:13:42.514462 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:13:42.514723 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:13:42.515371 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:13:42.515377 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:13:42.515681 (+ 304us) client_negotiation.cc:769] Sending connection context
0924 16:13:42.515941 (+ 260us) client_negotiation.cc:240] Negotiation successful
0924 16:13:42.516354 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":350,"threads_started":1}
I20260924 16:13:42.518762 9175 heartbeater.cc:347] Connected to a master server at 127.29.234.62:42321
I20260924 16:13:42.519119 9175 heartbeater.cc:464] Registering TS with master...
I20260924 16:13:42.519855 9175 heartbeater.cc:511] Master 127.29.234.62:42321 requested a full tablet report, sending...
I20260924 16:13:42.521248 8709 ts_manager.cc:194] Registered new tserver with Master: 06090350da8c44b1a413c98119175766 (127.29.234.3:44941)
I20260924 16:13:42.522063 8709 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.234.3@KRBTEST.COM'} at 127.29.234.3:43963
I20260924 16:13:42.527067 30632 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:13:42.543984 9041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:42.531313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52712 (local address 127.29.234.62:42321)
0924 16:13:42.531501 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:42.531505 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:42.531964 (+ 459us) server_negotiation.cc:407] Connection header received
0924 16:13:42.532149 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:42.532154 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:42.532229 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:42.532329 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:42.533720 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:42.534424 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:42.535644 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:42.536333 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:42.536498 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:42.538860 (+ 2362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:13:42.538866 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:13:42.538871 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:13:42.538910 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:13:42.541258 (+ 2348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:42.541978 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:42.541985 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:42.541989 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:42.542129 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:42.542504 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:42.542509 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:42.542513 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:42.542828 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:13:42.543101 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:42.543452 (+ 351us) server_negotiation.cc:299] Negotiation successful
0924 16:13:42.543673 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 16:13:42.548113 8709 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52712:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:13:42.555099 8709 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:13:42.593721 9191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:42.567789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:42169 (local address 127.0.0.1:42164)
0924 16:13:42.583788 (+ 15999us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:42.583814 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:42.583965 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:42.586163 (+ 2198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:42.586168 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:42.586196 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:13:42.586568 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:42.586581 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:42.588665 (+ 2084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:42.588670 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:42.590598 (+ 1928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:42.590611 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:42.592725 (+ 2114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:42.592732 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:42.592979 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:42.593045 (+ 66us) client_negotiation.cc:769] Sending connection context
0924 16:13:42.593175 (+ 130us) client_negotiation.cc:240] Negotiation successful
0924 16:13:42.593316 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15805,"mutex_wait_us":174,"thread_start_us":13995,"threads_started":1}
I20260924 16:13:42.593971 9192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:42.570149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38344 (local address 127.29.234.2:40583)
0924 16:13:42.583231 (+ 13082us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:42.583243 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:42.585918 (+ 2675us) server_negotiation.cc:407] Connection header received
0924 16:13:42.586174 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:42.586182 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:42.586390 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:42.586519 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:13:42.587218 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:42.588805 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:42.591371 (+ 2566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:42.592786 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:42.592926 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:42.593130 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:42.593475 (+ 345us) server_negotiation.cc:299] Negotiation successful
0924 16:13:42.593650 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12863,"thread_start_us":12719,"threads_started":1}
I20260924 16:13:42.594328 9188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:42.569934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.2:40583 (local address 127.0.0.1:38344)
0924 16:13:42.585771 (+ 15837us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:42.585788 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:42.585914 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:42.586744 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:42.586747 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:42.586766 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:13:42.587035 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:42.587042 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:42.588939 (+ 1897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:42.588944 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:42.591220 (+ 2276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:42.591229 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:42.593013 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:42.593019 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:42.593256 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:42.593281 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 16:13:42.593422 (+ 141us) client_negotiation.cc:240] Negotiation successful
0924 16:13:42.593550 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15688,"mutex_wait_us":25,"thread_start_us":13486,"threads_started":1}
I20260924 16:13:42.595019 9189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:42.567788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.3:44941 (local address 127.0.0.1:43710)
0924 16:13:42.581451 (+ 13663us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:42.581519 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:42.581747 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:42.585465 (+ 3718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:42.585475 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:42.585519 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:13:42.586053 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:42.586068 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:42.588295 (+ 2227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:42.588301 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:42.591006 (+ 2705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:42.591021 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:42.592771 (+ 1750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:42.592777 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:42.594237 (+ 1460us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:42.594271 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 16:13:42.594387 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 16:13:42.594663 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13280,"mutex_wait_us":905,"thread_start_us":17642,"threads_started":1}
I20260924 16:13:42.595446 9187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:42.567792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42164 (local address 127.29.234.1:42169)
0924 16:13:42.585515 (+ 17723us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:42.585525 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:42.585554 (+ 29us) server_negotiation.cc:407] Connection header received
0924 16:13:42.585652 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:42.585662 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:42.585910 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:42.586756 (+ 846us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:13:42.587034 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:42.588524 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:42.590757 (+ 2233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:42.592489 (+ 1732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:42.594666 (+ 2177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:42.594830 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:42.594881 (+ 51us) server_negotiation.cc:299] Negotiation successful
0924 16:13:42.594994 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17482,"thread_start_us":17286,"threads_started":1}
I20260924 16:13:42.595643 9190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:42.568294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43710 (local address 127.29.234.3:44941)
0924 16:13:42.584623 (+ 16329us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:42.584635 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:42.584680 (+ 45us) server_negotiation.cc:407] Connection header received
0924 16:13:42.584810 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:42.584822 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:42.585143 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:42.586049 (+ 906us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:13:42.586397 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:42.588141 (+ 1744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:42.591152 (+ 3011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:42.592542 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:42.592682 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:42.593026 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:42.594466 (+ 1440us) server_negotiation.cc:299] Negotiation successful
0924 16:13:42.595301 (+ 835us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16146,"thread_start_us":15978,"threads_started":1}
I20260924 16:13:42.598894 8835 tablet_service.cc:1467] Processing CreateTablet for tablet b2623f83754d4f39b4035983eba4a532 (DEFAULT_TABLE table=test-table [id=49a0e726166b4a42bcca362800bfbf95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:13:42.598852 8972 tablet_service.cc:1467] Processing CreateTablet for tablet b2623f83754d4f39b4035983eba4a532 (DEFAULT_TABLE table=test-table [id=49a0e726166b4a42bcca362800bfbf95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:13:42.599552 9108 tablet_service.cc:1467] Processing CreateTablet for tablet b2623f83754d4f39b4035983eba4a532 (DEFAULT_TABLE table=test-table [id=49a0e726166b4a42bcca362800bfbf95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:13:42.600219 8835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2623f83754d4f39b4035983eba4a532. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:13:42.600293 8972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2623f83754d4f39b4035983eba4a532. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:13:42.600868 9108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2623f83754d4f39b4035983eba4a532. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:13:42.612756 9193 tablet_bootstrap.cc:492] T b2623f83754d4f39b4035983eba4a532 P a913e5c7c99e40a4ba83897d03cb1108: Bootstrap starting.
I20260924 16:13:42.615736 9195 tablet_bootstrap.cc:492] T b2623f83754d4f39b4035983eba4a532 P 06090350da8c44b1a413c98119175766: Bootstrap starting.
I20260924 16:13:42.616235 9193 tablet_bootstrap.cc:654] T b2623f83754d4f39b4035983eba4a532 P a913e5c7c99e40a4ba83897d03cb1108: Neither blocks nor log segments found. Creating new log.
I20260924 16:13:42.617425 9193 log.cc:824] T b2623f83754d4f39b4035983eba4a532 P a913e5c7c99e40a4ba83897d03cb1108: Log is configured to *not* fsync() on all Append() calls
I20260924 16:13:42.619217 9194 tablet_bootstrap.cc:492] T b2623f83754d4f39b4035983eba4a532 P c005f1f00903408199b4debd9d2e1916: Bootstrap starting.
I20260924 16:13:42.620141 9195 tablet_bootstrap.cc:654] T b2623f83754d4f39b4035983eba4a532 P 06090350da8c44b1a413c98119175766: Neither blocks nor log segments found. Creating new log.
I20260924 16:13:42.621519 9195 log.cc:824] T b2623f83754d4f39b4035983eba4a532 P 06090350da8c44b1a413c98119175766: Log is configured to *not* fsync() on all Append() calls
I20260924 16:13:42.623152 9194 tablet_bootstrap.cc:654] T b2623f83754d4f39b4035983eba4a532 P c005f1f00903408199b4debd9d2e1916: Neither blocks nor log segments found. Creating new log.
I20260924 16:13:42.623806 9193 tablet_bootstrap.cc:492] T b2623f83754d4f39b4035983eba4a532 P a913e5c7c99e40a4ba83897d03cb1108: No bootstrap required, opened a new log
I20260924 16:13:42.624099 9193 ts_tablet_manager.cc:1397] T b2623f83754d4f39b4035983eba4a532 P a913e5c7c99e40a4ba83897d03cb1108: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.008s
I20260924 16:13:42.624274 9195 tablet_bootstrap.cc:492] T b2623f83754d4f39b4035983eba4a532 P 06090350da8c44b1a413c98119175766: No bootstrap required, opened a new log
I20260924 16:13:42.624333 9194 log.cc:824] T b2623f83754d4f39b4035983eba4a532 P c005f1f00903408199b4debd9d2e1916: Log is configured to *not* fsync() on all Append() calls
I20260924 16:13:42.624567 9195 ts_tablet_manager.cc:1397] T b2623f83754d4f39b4035983eba4a532 P 06090350da8c44b1a413c98119175766: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260924 16:13:42.627377 9194 tablet_bootstrap.cc:492] T b2623f83754d4f39b4035983eba4a532 P c005f1f00903408199b4debd9d2e1916: No bootstrap required, opened a new log
I20260924 16:13:42.627629 9194 ts_tablet_manager.cc:1397] T b2623f83754d4f39b4035983eba4a532 P c005f1f00903408199b4debd9d2e1916: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260924 16:13:42.632751 9195 raft_consensus.cc:348] T b2623f83754d4f39b4035983eba4a532 P 06090350da8c44b1a413c98119175766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a913e5c7c99e40a4ba83897d03cb1108" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 42169 } } peers { permanent_uuid: "06090350da8c44b1a413c98119175766" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 44941 } } peers { permanent_uuid: "c005f1f00903408199b4debd9d2e1916" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 40583 } }
I20260924 16:13:42.633118 9195 raft_consensus.cc:374] T b2623f83754d4f39b4035983eba4a532 P 06090350da8c44b1a413c98119175766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:13:42.633230 9195 raft_consensus.cc:729] T b2623f83754d4f39b4035983eba4a532 P 06090350da8c44b1a413c98119175766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06090350da8c44b1a413c98119175766, State: Initialized, Role: FOLLOWER
I20260924 16:13:42.633878 9195 consensus_queue.cc:260] T b2623f83754d4f39b4035983eba4a532 P 06090350da8c44b1a413c98119175766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a913e5c7c99e40a4ba83897d03cb1108" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 42169 } } peers { permanent_uuid: "06090350da8c44b1a413c98119175766" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 44941 } } peers { permanent_uuid: "c005f1f00903408199b4debd9d2e1916" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 40583 } }
I20260924 16:13:42.634639 9193 raft_consensus.cc:348] T b2623f83754d4f39b4035983eba4a532 P a913e5c7c99e40a4ba83897d03cb1108 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a913e5c7c99e40a4ba83897d03cb1108" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 42169 } } peers { permanent_uuid: "06090350da8c44b1a413c98119175766" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 44941 } } peers { permanent_uuid: "c005f1f00903408199b4debd9d2e1916" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 40583 } }
I20260924 16:13:42.634960 9175 heartbeater.cc:503] Master 127.29.234.62:42321 was elected leader, sending a full tablet report...
I20260924 16:13:42.635063 9193 raft_consensus.cc:374] T b2623f83754d4f39b4035983eba4a532 P a913e5c7c99e40a4ba83897d03cb1108 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:13:42.635147 9193 raft_consensus.cc:729] T b2623f83754d4f39b4035983eba4a532 P a913e5c7c99e40a4ba83897d03cb1108 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a913e5c7c99e40a4ba83897d03cb1108, State: Initialized, Role: FOLLOWER
I20260924 16:13:42.635648 9195 ts_tablet_manager.cc:1428] T b2623f83754d4f39b4035983eba4a532 P 06090350da8c44b1a413c98119175766: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260924 16:13:42.635767 9193 consensus_queue.cc:260] T b2623f83754d4f39b4035983eba4a532 P a913e5c7c99e40a4ba83897d03cb1108 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a913e5c7c99e40a4ba83897d03cb1108" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 42169 } } peers { permanent_uuid: "06090350da8c44b1a413c98119175766" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 44941 } } peers { permanent_uuid: "c005f1f00903408199b4debd9d2e1916" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 40583 } }
I20260924 16:13:42.637892 9193 ts_tablet_manager.cc:1428] T b2623f83754d4f39b4035983eba4a532 P a913e5c7c99e40a4ba83897d03cb1108: Time spent starting tablet: real 0.014s user 0.013s sys 0.002s
I20260924 16:13:42.638115 9194 raft_consensus.cc:348] T b2623f83754d4f39b4035983eba4a532 P c005f1f00903408199b4debd9d2e1916 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a913e5c7c99e40a4ba83897d03cb1108" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 42169 } } peers { permanent_uuid: "06090350da8c44b1a413c98119175766" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 44941 } } peers { permanent_uuid: "c005f1f00903408199b4debd9d2e1916" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 40583 } }
I20260924 16:13:42.638557 9194 raft_consensus.cc:374] T b2623f83754d4f39b4035983eba4a532 P c005f1f00903408199b4debd9d2e1916 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:13:42.638685 9194 raft_consensus.cc:729] T b2623f83754d4f39b4035983eba4a532 P c005f1f00903408199b4debd9d2e1916 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c005f1f00903408199b4debd9d2e1916, State: Initialized, Role: FOLLOWER
I20260924 16:13:42.639294 9194 consensus_queue.cc:260] T b2623f83754d4f39b4035983eba4a532 P c005f1f00903408199b4debd9d2e1916 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a913e5c7c99e40a4ba83897d03cb1108" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 42169 } } peers { permanent_uuid: "06090350da8c44b1a413c98119175766" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 44941 } } peers { permanent_uuid: "c005f1f00903408199b4debd9d2e1916" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 40583 } }
I20260924 16:13:42.640391 9039 heartbeater.cc:503] Master 127.29.234.62:42321 was elected leader, sending a full tablet report...
I20260924 16:13:42.641086 9194 ts_tablet_manager.cc:1428] T b2623f83754d4f39b4035983eba4a532 P c005f1f00903408199b4debd9d2e1916: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
I20260924 16:13:42.662098 9201 raft_consensus.cc:482] T b2623f83754d4f39b4035983eba4a532 P c005f1f00903408199b4debd9d2e1916 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:13:42.662434 9201 raft_consensus.cc:504] T b2623f83754d4f39b4035983eba4a532 P c005f1f00903408199b4debd9d2e1916 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a913e5c7c99e40a4ba83897d03cb1108" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 42169 } } peers { permanent_uuid: "06090350da8c44b1a413c98119175766" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 44941 } } peers { permanent_uuid: "c005f1f00903408199b4debd9d2e1916" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 40583 } }
I20260924 16:13:42.664206 9201 leader_election.cc:288] T b2623f83754d4f39b4035983eba4a532 P c005f1f00903408199b4debd9d2e1916 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a913e5c7c99e40a4ba83897d03cb1108 (127.29.234.1:42169), 06090350da8c44b1a413c98119175766 (127.29.234.3:44941)
I20260924 16:13:42.674082 9190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:42.664788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.2:46315 (local address 127.29.234.3:44941)
0924 16:13:42.665012 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:42.665020 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:42.665880 (+ 860us) server_negotiation.cc:407] Connection header received
0924 16:13:42.666072 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:42.666078 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:42.666177 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:42.666289 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:13:42.667174 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:42.668375 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:42.671512 (+ 3137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:42.672836 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:42.673005 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:42.673084 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:42.673690 (+ 606us) server_negotiation.cc:299] Negotiation successful
0924 16:13:42.673828 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 16:13:42.674082 9187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:42.664857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.2:52929 (local address 127.29.234.1:42169)
0924 16:13:42.665050 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:42.665057 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:42.665860 (+ 803us) server_negotiation.cc:407] Connection header received
0924 16:13:42.666072 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:42.666078 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:42.666223 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:42.666316 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:13:42.667270 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:42.668366 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:42.671514 (+ 3148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:42.672834 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:42.673005 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:42.673087 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:42.673689 (+ 602us) server_negotiation.cc:299] Negotiation successful
0924 16:13:42.673828 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 16:13:42.674319 9202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:42.664779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:42169 (local address 127.29.234.2:52929)
0924 16:13:42.665705 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:42.665724 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:42.665915 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:42.666342 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:42.666347 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:42.666379 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:13:42.667104 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:42.667116 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:42.668509 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:42.668512 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:42.671329 (+ 2817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:42.671344 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:42.673128 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:42.673134 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:42.673468 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:42.673526 (+ 58us) client_negotiation.cc:769] Sending connection context
0924 16:13:42.673634 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 16:13:42.673803 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":761,"mutex_wait_us":364,"spinlock_wait_cycles":22528,"thread_start_us":475,"threads_started":1}
I20260924 16:13:42.674588 9203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:42.664809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.3:44941 (local address 127.29.234.2:46315)
0924 16:13:42.665759 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0924 16:13:42.665775 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:13:42.665915 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:13:42.666342 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:13:42.666358 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:13:42.666386 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:13:42.666970 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:42.666984 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:42.668507 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:42.668512 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:42.671329 (+ 2817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:13:42.671344 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:13:42.673127 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:13:42.673134 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:13:42.673474 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:42.673528 (+ 54us) client_negotiation.cc:769] Sending connection context
0924 16:13:42.673634 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 16:13:42.673803 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":845,"mutex_wait_us":368,"spinlock_wait_cycles":40704,"thread_start_us":525,"threads_started":1}
I20260924 16:13:42.675565 8857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2623f83754d4f39b4035983eba4a532" candidate_uuid: "c005f1f00903408199b4debd9d2e1916" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a913e5c7c99e40a4ba83897d03cb1108" is_pre_election: true
I20260924 16:13:42.675570 9128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2623f83754d4f39b4035983eba4a532" candidate_uuid: "c005f1f00903408199b4debd9d2e1916" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06090350da8c44b1a413c98119175766" is_pre_election: true
I20260924 16:13:42.676174 9128 raft_consensus.cc:2454] T b2623f83754d4f39b4035983eba4a532 P 06090350da8c44b1a413c98119175766 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c005f1f00903408199b4debd9d2e1916 in term 0.
I20260924 16:13:42.676174 8857 raft_consensus.cc:2454] T b2623f83754d4f39b4035983eba4a532 P a913e5c7c99e40a4ba83897d03cb1108 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c005f1f00903408199b4debd9d2e1916 in term 0.
I20260924 16:13:42.677003 8926 leader_election.cc:302] T b2623f83754d4f39b4035983eba4a532 P c005f1f00903408199b4debd9d2e1916 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 06090350da8c44b1a413c98119175766, a913e5c7c99e40a4ba83897d03cb1108, c005f1f00903408199b4debd9d2e1916; no voters:
I20260924 16:13:42.677500 9201 raft_consensus.cc:2781] T b2623f83754d4f39b4035983eba4a532 P c005f1f00903408199b4debd9d2e1916 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:13:42.677656 9201 raft_consensus.cc:482] T b2623f83754d4f39b4035983eba4a532 P c005f1f00903408199b4debd9d2e1916 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:13:42.677778 9201 raft_consensus.cc:3037] T b2623f83754d4f39b4035983eba4a532 P c005f1f00903408199b4debd9d2e1916 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:13:42.680084 9201 raft_consensus.cc:504] T b2623f83754d4f39b4035983eba4a532 P c005f1f00903408199b4debd9d2e1916 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a913e5c7c99e40a4ba83897d03cb1108" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 42169 } } peers { permanent_uuid: "06090350da8c44b1a413c98119175766" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 44941 } } peers { permanent_uuid: "c005f1f00903408199b4debd9d2e1916" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 40583 } }
I20260924 16:13:42.680789 9201 leader_election.cc:288] T b2623f83754d4f39b4035983eba4a532 P c005f1f00903408199b4debd9d2e1916 [CANDIDATE]: Term 1 election: Requested vote from peers a913e5c7c99e40a4ba83897d03cb1108 (127.29.234.1:42169), 06090350da8c44b1a413c98119175766 (127.29.234.3:44941)
I20260924 16:13:42.681198 8857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2623f83754d4f39b4035983eba4a532" candidate_uuid: "c005f1f00903408199b4debd9d2e1916" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a913e5c7c99e40a4ba83897d03cb1108"
I20260924 16:13:42.681432 8857 raft_consensus.cc:3037] T b2623f83754d4f39b4035983eba4a532 P a913e5c7c99e40a4ba83897d03cb1108 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:13:42.681339 9128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2623f83754d4f39b4035983eba4a532" candidate_uuid: "c005f1f00903408199b4debd9d2e1916" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06090350da8c44b1a413c98119175766"
I20260924 16:13:42.681613 9128 raft_consensus.cc:3037] T b2623f83754d4f39b4035983eba4a532 P 06090350da8c44b1a413c98119175766 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:13:42.683753 8857 raft_consensus.cc:2454] T b2623f83754d4f39b4035983eba4a532 P a913e5c7c99e40a4ba83897d03cb1108 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c005f1f00903408199b4debd9d2e1916 in term 1.
I20260924 16:13:42.684145 9128 raft_consensus.cc:2454] T b2623f83754d4f39b4035983eba4a532 P 06090350da8c44b1a413c98119175766 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c005f1f00903408199b4debd9d2e1916 in term 1.
I20260924 16:13:42.684281 8924 leader_election.cc:302] T b2623f83754d4f39b4035983eba4a532 P c005f1f00903408199b4debd9d2e1916 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a913e5c7c99e40a4ba83897d03cb1108, c005f1f00903408199b4debd9d2e1916; no voters:
I20260924 16:13:42.684579 9201 raft_consensus.cc:2781] T b2623f83754d4f39b4035983eba4a532 P c005f1f00903408199b4debd9d2e1916 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:13:42.685052 9201 raft_consensus.cc:686] T b2623f83754d4f39b4035983eba4a532 P c005f1f00903408199b4debd9d2e1916 [term 1 LEADER]: Becoming Leader. State: Replica: c005f1f00903408199b4debd9d2e1916, State: Running, Role: LEADER
I20260924 16:13:42.685609 9201 consensus_queue.cc:237] T b2623f83754d4f39b4035983eba4a532 P c005f1f00903408199b4debd9d2e1916 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a913e5c7c99e40a4ba83897d03cb1108" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 42169 } } peers { permanent_uuid: "06090350da8c44b1a413c98119175766" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 44941 } } peers { permanent_uuid: "c005f1f00903408199b4debd9d2e1916" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 40583 } }
I20260924 16:13:42.690428 8709 catalog_manager.cc:5495] T b2623f83754d4f39b4035983eba4a532 P c005f1f00903408199b4debd9d2e1916 reported cstate change: term changed from 0 to 1, leader changed from <none> to c005f1f00903408199b4debd9d2e1916 (127.29.234.2). New cstate: current_term: 1 leader_uuid: "c005f1f00903408199b4debd9d2e1916" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a913e5c7c99e40a4ba83897d03cb1108" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 42169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06090350da8c44b1a413c98119175766" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 44941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c005f1f00903408199b4debd9d2e1916" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 40583 } health_report { overall_health: HEALTHY } } }
W20260924 16:13:42.706581 8903 tablet.cc:2369] T b2623f83754d4f39b4035983eba4a532 P a913e5c7c99e40a4ba83897d03cb1108: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:13:42.716204 9192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:42.711239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38360 (local address 127.29.234.2:40583)
0924 16:13:42.711435 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:42.711439 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:42.711456 (+ 17us) server_negotiation.cc:407] Connection header received
0924 16:13:42.711540 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:42.711544 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:42.711617 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:42.711738 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:13:42.712060 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:42.712809 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:42.714200 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:42.714662 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:42.714784 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:42.715204 (+ 420us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:13:42.715713 (+ 509us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:13:42.715844 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:42.715928 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 16:13:42.715993 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 16:13:42.724699 9128 raft_consensus.cc:1261] T b2623f83754d4f39b4035983eba4a532 P 06090350da8c44b1a413c98119175766 [term 1 FOLLOWER]: Refusing update from remote peer c005f1f00903408199b4debd9d2e1916: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:13:42.724747 8857 raft_consensus.cc:1261] T b2623f83754d4f39b4035983eba4a532 P a913e5c7c99e40a4ba83897d03cb1108 [term 1 FOLLOWER]: Refusing update from remote peer c005f1f00903408199b4debd9d2e1916: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:13:42.725742 9204 consensus_queue.cc:1035] T b2623f83754d4f39b4035983eba4a532 P c005f1f00903408199b4debd9d2e1916 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a913e5c7c99e40a4ba83897d03cb1108" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 42169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:13:42.726110 9201 consensus_queue.cc:1035] T b2623f83754d4f39b4035983eba4a532 P c005f1f00903408199b4debd9d2e1916 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06090350da8c44b1a413c98119175766" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 44941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 16:13:42.729055 9040 tablet.cc:2369] T b2623f83754d4f39b4035983eba4a532 P c005f1f00903408199b4debd9d2e1916: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:13:42.740273 9209 mvcc.cc:205] Tried to move back new op lower bound from 7332931267470434304 to 7332931267324338176. Current Snapshot: MvccSnapshot[applied={T|T < 7332931267470434304}]
I20260924 16:13:42.758157 8708 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52712:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:13:42.758464 8708 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52712:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:13:42.762449 8708 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 36d0d7e738254dfc906c44c86c75166e: Sending DeleteTablet for 3 replicas of tablet b2623f83754d4f39b4035983eba4a532
I20260924 16:13:42.763592 8835 tablet_service.cc:1514] Processing DeleteTablet for tablet b2623f83754d4f39b4035983eba4a532 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:13:42 UTC) from {username='kudu', principal='kudu/127.29.234.62@KRBTEST.COM'} at 127.0.0.1:42164
I20260924 16:13:42.763968 8972 tablet_service.cc:1514] Processing DeleteTablet for tablet b2623f83754d4f39b4035983eba4a532 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:13:42 UTC) from {username='kudu', principal='kudu/127.29.234.62@KRBTEST.COM'} at 127.0.0.1:38344
I20260924 16:13:42.764000 9108 tablet_service.cc:1514] Processing DeleteTablet for tablet b2623f83754d4f39b4035983eba4a532 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:13:42 UTC) from {username='kudu', principal='kudu/127.29.234.62@KRBTEST.COM'} at 127.0.0.1:43710
I20260924 16:13:42.766086 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 8770
I20260924 16:13:42.778824 9219 tablet_replica.cc:330] T b2623f83754d4f39b4035983eba4a532 P a913e5c7c99e40a4ba83897d03cb1108: stopping tablet replica
I20260924 16:13:42.779558 9220 tablet_replica.cc:330] T b2623f83754d4f39b4035983eba4a532 P c005f1f00903408199b4debd9d2e1916: stopping tablet replica
I20260924 16:13:42.780236 9221 tablet_replica.cc:330] T b2623f83754d4f39b4035983eba4a532 P 06090350da8c44b1a413c98119175766: stopping tablet replica
I20260924 16:13:42.780539 9219 raft_consensus.cc:2229] T b2623f83754d4f39b4035983eba4a532 P a913e5c7c99e40a4ba83897d03cb1108 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:13:42.781055 9220 raft_consensus.cc:2229] T b2623f83754d4f39b4035983eba4a532 P c005f1f00903408199b4debd9d2e1916 [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:13:42.781136 9219 raft_consensus.cc:2258] T b2623f83754d4f39b4035983eba4a532 P a913e5c7c99e40a4ba83897d03cb1108 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:13:42.781355 9221 raft_consensus.cc:2229] T b2623f83754d4f39b4035983eba4a532 P 06090350da8c44b1a413c98119175766 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:13:42.781911 9221 raft_consensus.cc:2258] T b2623f83754d4f39b4035983eba4a532 P 06090350da8c44b1a413c98119175766 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:13:42.782583 9220 raft_consensus.cc:2258] T b2623f83754d4f39b4035983eba4a532 P c005f1f00903408199b4debd9d2e1916 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:13:42.783726 9219 ts_tablet_manager.cc:1905] T b2623f83754d4f39b4035983eba4a532 P a913e5c7c99e40a4ba83897d03cb1108: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:13:42.784790 9220 ts_tablet_manager.cc:1905] T b2623f83754d4f39b4035983eba4a532 P c005f1f00903408199b4debd9d2e1916: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:13:42.785871 9221 ts_tablet_manager.cc:1905] T b2623f83754d4f39b4035983eba4a532 P 06090350da8c44b1a413c98119175766: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:13:42.790824 9219 ts_tablet_manager.cc:1918] T b2623f83754d4f39b4035983eba4a532 P a913e5c7c99e40a4ba83897d03cb1108: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:13:42.791004 9219 log.cc:1196] T b2623f83754d4f39b4035983eba4a532 P a913e5c7c99e40a4ba83897d03cb1108: Deleting WAL directory at /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/minicluster-data/ts-0/wal/wals/b2623f83754d4f39b4035983eba4a532
I20260924 16:13:42.791597 9219 ts_tablet_manager.cc:1939] T b2623f83754d4f39b4035983eba4a532 P a913e5c7c99e40a4ba83897d03cb1108: Deleting consensus metadata
Sep 24 16:13:42 dist-test-slave-v5j9 krb5kdc[8662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266420, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.234.1@KRBTEST.COM
I20260924 16:13:42.792738 9220 ts_tablet_manager.cc:1918] T b2623f83754d4f39b4035983eba4a532 P c005f1f00903408199b4debd9d2e1916: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:13:42.793068 9220 log.cc:1196] T b2623f83754d4f39b4035983eba4a532 P c005f1f00903408199b4debd9d2e1916: Deleting WAL directory at /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/minicluster-data/ts-1/wal/wals/b2623f83754d4f39b4035983eba4a532
I20260924 16:13:42.793560 8692 catalog_manager.cc:4841] TS a913e5c7c99e40a4ba83897d03cb1108 (127.29.234.1:42169): tablet b2623f83754d4f39b4035983eba4a532 (table test-table [id=49a0e726166b4a42bcca362800bfbf95]) successfully deleted
I20260924 16:13:42.794222 9220 ts_tablet_manager.cc:1939] T b2623f83754d4f39b4035983eba4a532 P c005f1f00903408199b4debd9d2e1916: Deleting consensus metadata
I20260924 16:13:42.795944 8695 catalog_manager.cc:4841] TS c005f1f00903408199b4debd9d2e1916 (127.29.234.2:40583): tablet b2623f83754d4f39b4035983eba4a532 (table test-table [id=49a0e726166b4a42bcca362800bfbf95]) successfully deleted
I20260924 16:13:42.796133 9221 ts_tablet_manager.cc:1918] T b2623f83754d4f39b4035983eba4a532 P 06090350da8c44b1a413c98119175766: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:13:42.796317 9221 log.cc:1196] T b2623f83754d4f39b4035983eba4a532 P 06090350da8c44b1a413c98119175766: Deleting WAL directory at /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790266323169505-30632-0/minicluster-data/ts-2/wal/wals/b2623f83754d4f39b4035983eba4a532
I20260924 16:13:42.796756 9221 ts_tablet_manager.cc:1939] T b2623f83754d4f39b4035983eba4a532 P 06090350da8c44b1a413c98119175766: Deleting consensus metadata
I20260924 16:13:42.798228 8693 catalog_manager.cc:4841] TS 06090350da8c44b1a413c98119175766 (127.29.234.3:44941): tablet b2623f83754d4f39b4035983eba4a532 (table test-table [id=49a0e726166b4a42bcca362800bfbf95]) successfully deleted
I20260924 16:13:42.801227 9187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:42.767584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42178 (local address 127.29.234.1:42169)
0924 16:13:42.767756 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:42.767763 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:42.772216 (+ 4453us) server_negotiation.cc:407] Connection header received
0924 16:13:42.772427 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:42.772432 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:42.772514 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:42.772606 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:42.775187 (+ 2581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:42.776453 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:42.781056 (+ 4603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:42.781812 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:42.781941 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:42.794281 (+ 12340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:13:42.794295 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:13:42.794316 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:13:42.794367 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:13:42.798278 (+ 3911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:42.799204 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:42.799213 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:42.799215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:42.799354 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:42.799799 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:42.799803 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:42.799805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:42.800163 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:13:42.800391 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:42.800735 (+ 344us) server_negotiation.cc:299] Negotiation successful
0924 16:13:42.800917 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 16:13:42.802013 8897 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:13:42.931254 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 8770
I20260924 16:13:42.950187 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 8908
Sep 24 16:13:42 dist-test-slave-v5j9 krb5kdc[8662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266420, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.234.2@KRBTEST.COM
I20260924 16:13:42.968164 9192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:42.951001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38374 (local address 127.29.234.2:40583)
0924 16:13:42.951194 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:42.951199 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:42.951255 (+ 56us) server_negotiation.cc:407] Connection header received
0924 16:13:42.951468 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:42.951477 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:42.951558 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:42.951639 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:42.953197 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:42.954117 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:42.955968 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:42.956381 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:42.956472 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:42.961638 (+ 5166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:13:42.961661 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:13:42.961690 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:13:42.961739 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:13:42.964949 (+ 3210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:42.965889 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:42.965902 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:42.965905 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:42.966115 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:42.966612 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:42.966619 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:42.966623 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:42.967104 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:13:42.967396 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:42.967676 (+ 280us) server_negotiation.cc:299] Negotiation successful
0924 16:13:42.967851 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 16:13:42.968855 9034 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:13:43.084088 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 8908
I20260924 16:13:43.103041 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 9045
Sep 24 16:13:43 dist-test-slave-v5j9 krb5kdc[8662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266420, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.234.3@KRBTEST.COM
I20260924 16:13:43.117477 9190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:13:43.103772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43724 (local address 127.29.234.3:44941)
0924 16:13:43.103934 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 16:13:43.103939 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:13:43.103959 (+ 20us) server_negotiation.cc:407] Connection header received
0924 16:13:43.104143 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:13:43.104147 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:13:43.104209 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:13:43.104293 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:13:43.105570 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:43.106279 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:43.107450 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:13:43.107813 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:13:43.107899 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:13:43.112139 (+ 4240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:13:43.112151 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:13:43.112167 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:13:43.112202 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:13:43.115090 (+ 2888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:43.115819 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:43.115826 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:43.115828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:43.115964 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:13:43.116252 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:13:43.116255 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:13:43.116257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:13:43.116594 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:13:43.116802 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0924 16:13:43.117004 (+ 202us) server_negotiation.cc:299] Negotiation successful
0924 16:13:43.117170 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 16:13:43.118083 9170 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:13:43.206714 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 9045
I20260924 16:13:43.223730 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 8676
I20260924 16:13:43.224861 8739 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:13:43.335510 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 8676
2026-09-24T16:13:43Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4315 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:13:43 dist-test-slave-v5j9 krb5kdc[9229](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:13:43 dist-test-slave-v5j9 krb5kdc[9229](info): set up 2 sockets
Sep 24 16:13:43 dist-test-slave-v5j9 krb5kdc[9229](info): commencing operation
krb5kdc: starting...
W20260924 16:13:44.462306 30632 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.008s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Sep 24 16:13:44 dist-test-slave-v5j9 krb5kdc[9229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:13:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:13:44Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.29.234.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.29.234.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.29.234.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/krb5kdc/rangeradmin_127.29.234.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.29.234.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/krb5kdc/rangeradmin_127.29.234.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.29.234.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.29.234.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.29.234.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/krb5kdc/rangerlookup_127.29.234.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.29.234.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/krb5kdc/rangerlookup_127.29.234.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.29.234.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.29.234.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.29.234.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/krb5kdc/HTTP_127.29.234.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.29.234.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/krb5kdc/HTTP_127.29.234.20@KRBTEST.COM.keytab.
I20260924 16:13:44.790517 30632 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskQurfSc/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/postgres -l logfile start
2026-09-24 16:13:47.315 UTC [9259] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 16:13:47.315 UTC [9259] LOG: listening on IPv4 address "127.29.234.20", port 40939
2026-09-24 16:13:47.319 UTC [9259] LOG: listening on Unix socket "/tmp/.s.PGSQL.40939"
2026-09-24 16:13:47.340 UTC [9262] LOG: database system was shut down at 2026-09-24 16:13:45 UTC
2026-09-24 16:13:47.343 UTC [9259] LOG: database system is ready to accept connections
I20260924 16:13:48.321269 30632 mini_postgres.cc:96] Postgres bound to 40939
2026-09-24 16:13:48.340 UTC [9270] FATAL: database "slave" does not exist
127.29.234.20:40939 - accepting connections
I20260924 16:13:48.341177 30632 mini_ranger.cc:161] Starting Ranger...
I20260924 16:13:48.369624 30632 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 16:13:48.865656 30632 mini_ranger.cc:87] Created ranger Postgres database
I20260924 16:13:48.865778 30632 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/ranger-admin
2026-09-24 16:13:49,232 [I] DB FLAVOR :POSTGRES
2026-09-24 16:13:49,232 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 16:13:49,233 [I] Checking connection..
2026-09-24 16:13:49,233 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQurfSc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQurfSc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.234.20:40939/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 16:13:49,501 [I] Checking connection passed.
2026-09-24 16:13:49,501 [I] --------- Verifying version history table ---------
2026-09-24 16:13:49,501 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQurfSc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQurfSc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.234.20:40939/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 16:13:49,733 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 16:13:49,733 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQurfSc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQurfSc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.234.20:40939/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 16:13:49,987 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 16:13:49,987 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 16:13:49,988 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQurfSc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQurfSc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.234.20:40939/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskQurfSc/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 16:13:50.242 UTC [9337] WARNING: there is no transaction in progress
2026-09-24 16:13:50,249 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 16:13:50,249 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQurfSc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQurfSc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.234.20:40939/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 16:13:50,512 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 16:13:50,512 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 16:13:50,512 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQurfSc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQurfSc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.234.20:40939/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 16:13:50,758 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQurfSc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQurfSc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.234.20:40939/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 16:13:51,025 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQurfSc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQurfSc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.234.20:40939/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-v5j9.c.gcp-upstream.internal','N') ;"
2026-09-24 16:13:51,265 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQurfSc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQurfSc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.234.20:40939/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 16:13:51,517 [I] Table x_portal_user does not exist in database ranger
2026-09-24 16:13:51,518 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQurfSc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQurfSc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.234.20:40939/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 16:13:51,758 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 16:13:51,758 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQurfSc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQurfSc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.234.20:40939/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 16:13:52,001 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQurfSc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQurfSc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.234.20:40939/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 16:13:52,258 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 16:13:52,258 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQurfSc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQurfSc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.234.20:40939/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskQurfSc/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 16:13:53.215 UTC [9446] WARNING: there is no transaction in progress
2026-09-24 16:13:53.231 UTC [9446] WARNING: there is no transaction in progress
2026-09-24 16:13:53.247 UTC [9446] WARNING: there is no transaction in progress
2026-09-24 16:13:53.262 UTC [9446] WARNING: there is no transaction in progress
2026-09-24 16:13:53.277 UTC [9446] WARNING: there is no transaction in progress
2026-09-24 16:13:53.294 UTC [9446] WARNING: there is no transaction in progress
2026-09-24 16:13:53.352 UTC [9446] WARNING: there is no transaction in progress
2026-09-24 16:13:53.359 UTC [9446] WARNING: there is no transaction in progress
2026-09-24 16:13:53.374 UTC [9446] WARNING: there is no transaction in progress
2026-09-24 16:13:53.386 UTC [9446] WARNING: there is no transaction in progress
2026-09-24 16:13:53.397 UTC [9446] WARNING: there is no transaction in progress
2026-09-24 16:13:53.411 UTC [9446] WARNING: there is no transaction in progress
2026-09-24 16:13:53.421 UTC [9446] WARNING: there is no transaction in progress
2026-09-24 16:13:53.428 UTC [9446] WARNING: there is no transaction in progress
2026-09-24 16:13:53.435 UTC [9446] WARNING: there is no transaction in progress
2026-09-24 16:13:53.442 UTC [9446] WARNING: there is no transaction in progress
2026-09-24 16:13:54,095 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 16:13:54,095 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQurfSc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQurfSc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.234.20:40939/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 16:13:54,336 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 16:13:54,337 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQurfSc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQurfSc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.234.20:40939/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 16:13:54,576 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 16:13:54,577 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQurfSc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQurfSc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.234.20:40939/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 16:13:54,840 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 16:13:54,840 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQurfSc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQurfSc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.234.20:40939/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 16:13:55,066 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQurfSc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQurfSc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.234.20:40939/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 16:13:55,322 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQurfSc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQurfSc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.234.20:40939/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-v5j9.c.gcp-upstream.internal';"
2026-09-24 16:13:55,549 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 16:13:55,549 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQurfSc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQurfSc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.234.20:40939/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 16:13:55,784 [I] DB_PATCHES have already been applied
I20260924 16:13:55.792013 30632 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/ranger-admin:/tmp/dist-test-taskQurfSc/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskQurfSc/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskQurfSc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQurfSc/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 16:13:55.792107 30632 mini_ranger.cc:193] Using host: 127.29.234.20
I20260924 16:13:55.807752 30632 mini_ranger.cc:227] Ranger admin URL: http://127.29.234.20:36513
Sep 24, 2026 4:13:56 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 4:13:56 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 4:13:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 4:13:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 4:13:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 4:13:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 4:13:56 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 16:13:56.968387 30632 mini_ranger.cc:160] Time spent starting Ranger: real 8.627s user 0.006s sys 0.035s
Sep 24, 2026 4:14:16 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 4:14:16 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 24, 2026 4:14:16 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 4:14:16 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 4:14:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 4:14:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 4:14:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 4:14:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 4:14:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 4:14:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 4:14:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 4:14:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 4:14:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 4:14:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 4:14:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 4:14:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 4:14:16 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 4:14:17 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 16:14:17.678581 30632 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.29.234.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.29.234.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.29.234.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/krb5kdc/rangerkms_127.29.234.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.29.234.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/krb5kdc/rangerkms_127.29.234.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.29.234.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.29.234.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.29.234.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/krb5kdc/HTTP_127.29.234.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.29.234.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/krb5kdc/HTTP_127.29.234.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 16:14:17 dist-test-slave-v5j9 krb5kdc[9229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266457, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 16:14:17.880435 30632 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 16:14:17.904436 30632 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 16:14:18.820323 30632 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 16:14:18.820465 30632 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/ranger-kms
I20260924 16:14:18.820639 30632 mini_ranger_kms.cc:227] Using postgres at 127.29.234.20:40939
2026-09-24 16:14:19,659 [I] DB FLAVOR :POSTGRES
2026-09-24 16:14:19,659 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 16:14:19,659 [I] Checking connection
2026-09-24 16:14:19,659 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQurfSc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.234.20:40939/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 16:14:19,909 [I] connection success
2026-09-24 16:14:19,909 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 16:14:19,910 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 16:14:19,910 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQurfSc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.234.20:40939/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 16:14:20,141 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 16:14:20,141 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 16:14:20,141 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 16:14:20,141 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQurfSc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.234.20:40939/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 16:14:20,423 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 16:14:20.907471 30632 mini_ranger_kms.cc:310] Created kms service
I20260924 16:14:21.118191 30632 mini_ranger_kms.cc:326] Created kudu user
I20260924 16:14:21.163563 30632 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 16:14:21 dist-test-slave-v5j9 krb5kdc[9229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266457, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.29.234.20@KRBTEST.COM
I20260924 16:14:21.586715 30632 mini_ranger_kms.cc:384] Added ranger policy
I20260924 16:14:21.587050 30632 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/ranger-kms:/tmp/dist-test-taskQurfSc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQurfSc/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskQurfSc/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 16:14:21.587108 30632 mini_ranger_kms.cc:240] Using host: 127.29.234.20
I20260924 16:14:21.608892 30632 mini_ranger_kms.cc:276] Ranger KMS PID: 9629
I20260924 16:14:21.609098 30632 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.29.234.20:45061
Sep 24, 2026 4:14:22 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 4:14:22 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 4:14:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 4:14:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 4:14:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 4:14:23 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 16:14:23.935935 30632 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.056s user 0.165s sys 0.377s
I20260924 16:14:23.936192 30632 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 16:14:23.936419 30632 mini_ranger_kms.cc:401] 127.29.234.20:45061/kms/v1/keys
Sep 24 16:14:24 dist-test-slave-v5j9 krb5kdc[9229](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790266464, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.29.234.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 16:14:24 dist-test-slave-v5j9 krb5kdc[9229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266464, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.29.234.20@KRBTEST.COM for HTTP/127.29.234.20@KRBTEST.COM
Sep 24 16:14:24 dist-test-slave-v5j9 krb5kdc[9229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266464, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.29.234.20@KRBTEST.COM for HTTP/127.29.234.20@KRBTEST.COM
Sep 24 16:14:27 dist-test-slave-v5j9 krb5kdc[9229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.29.234.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab.
Entry for principal kudu/127.29.234.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab.
I20260924 16:14:27.205694 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:33493
--webserver_interface=127.29.234.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab
--principal=kudu/127.29.234.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:44353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.234.62:33493
--ranger_config_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.234.20:45061/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:14:27.572768 9682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:14:27.573078 9682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:14:27.573222 9682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:14:27.580559 9682 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:14:27.580681 9682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:14:27.580783 9682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:14:27.580883 9682 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:14:27.580973 9682 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:14:27.591687 9682 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:44353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/master-0/wal
--ranger_kms_url=127.29.234.20:45061/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.29.234.62:33493
--ranger_config_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab
--principal=kudu/127.29.234.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:33493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.234.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:14:27.593151 9682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:14:27.594523 9682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:14:27.610026 9687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:14:27.610482 9691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:14:27.610116 9689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:14:27.612576 9682 server_base.cc:1034] running on GCE node
I20260924 16:14:27.613600 9682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:14:27.615438 9682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:14:27.616693 9682 hybrid_clock.cc:648] HybridClock initialized: now 1790266467616634 us; error 56 us; skew 500 ppm
I20260924 16:14:27.617074 9682 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:14:27 dist-test-slave-v5j9 krb5kdc[9229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266467, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:14:27.626577 9682 init.cc:377] Logged in from keytab as kudu/127.29.234.62@KRBTEST.COM (short username kudu)
I20260924 16:14:27.628504 9682 webserver.cc:460] Webserver started at http://127.29.234.62:34403/ using document root <none> and password file <none>
I20260924 16:14:27.629276 9682 fs_manager.cc:359] Metadata directory not provided
I20260924 16:14:27.629475 9682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:14:27.629842 9682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 16:14:27 dist-test-slave-v5j9 krb5kdc[9229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266467, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.62@KRBTEST.COM for HTTP/127.29.234.20@KRBTEST.COM
I20260924 16:14:27.683709 9682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/master-0/data/instance:
uuid: "bace56cc062f4d4bb51afd00f6eba991"
format_stamp: "Formatted at 2026-09-24 16:14:27 on dist-test-slave-v5j9"
server_key: "c0bea2d4d0c648ece6d8efa8a852e1df"
server_key_iv: "74787d8351a78f17bddb8e3608f947d9"
server_key_version: "kuduclusterkey@0"
I20260924 16:14:27.684839 9682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/master-0/wal/instance:
uuid: "bace56cc062f4d4bb51afd00f6eba991"
format_stamp: "Formatted at 2026-09-24 16:14:27 on dist-test-slave-v5j9"
server_key: "c0bea2d4d0c648ece6d8efa8a852e1df"
server_key_iv: "74787d8351a78f17bddb8e3608f947d9"
server_key_version: "kuduclusterkey@0"
I20260924 16:14:27.691967 9682 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20260924 16:14:27.713150 9698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:14:27.715142 9682 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 16:14:27.715363 9682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/master-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/master-0/wal
uuid: "bace56cc062f4d4bb51afd00f6eba991"
format_stamp: "Formatted at 2026-09-24 16:14:27 on dist-test-slave-v5j9"
server_key: "c0bea2d4d0c648ece6d8efa8a852e1df"
server_key_iv: "74787d8351a78f17bddb8e3608f947d9"
server_key_version: "kuduclusterkey@0"
I20260924 16:14:27.715643 9682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:14:27.737195 9682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:14:27.742450 9682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:14:27.742770 9682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:14:27.772361 9682 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.62:33493
I20260924 16:14:27.772364 9749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.62:33493 every 8 connection(s)
I20260924 16:14:27.774286 9682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
I20260924 16:14:27.776543 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 9682
I20260924 16:14:27.776774 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/master-0/wal/instance
I20260924 16:14:27.777684 30632 external_mini_cluster.cc:1428] Setting key ea9488fefaec62c6ccf2c5828278cbf5
I20260924 16:14:27.779166 9750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:14:27.798285 9750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bace56cc062f4d4bb51afd00f6eba991: Bootstrap starting.
I20260924 16:14:27.802804 9750 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bace56cc062f4d4bb51afd00f6eba991: Neither blocks nor log segments found. Creating new log.
I20260924 16:14:27.804196 9750 log.cc:824] T 00000000000000000000000000000000 P bace56cc062f4d4bb51afd00f6eba991: Log is configured to *not* fsync() on all Append() calls
I20260924 16:14:27.808760 9750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bace56cc062f4d4bb51afd00f6eba991: No bootstrap required, opened a new log
Sep 24 16:14:27 dist-test-slave-v5j9 krb5kdc[9229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:14:27.818331 9750 raft_consensus.cc:348] T 00000000000000000000000000000000 P bace56cc062f4d4bb51afd00f6eba991 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bace56cc062f4d4bb51afd00f6eba991" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 33493 } }
I20260924 16:14:27.818845 9750 raft_consensus.cc:374] T 00000000000000000000000000000000 P bace56cc062f4d4bb51afd00f6eba991 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:14:27.818994 9750 raft_consensus.cc:729] T 00000000000000000000000000000000 P bace56cc062f4d4bb51afd00f6eba991 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bace56cc062f4d4bb51afd00f6eba991, State: Initialized, Role: FOLLOWER
I20260924 16:14:27.819676 9750 consensus_queue.cc:260] T 00000000000000000000000000000000 P bace56cc062f4d4bb51afd00f6eba991 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bace56cc062f4d4bb51afd00f6eba991" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 33493 } }
I20260924 16:14:27.819921 9750 raft_consensus.cc:388] T 00000000000000000000000000000000 P bace56cc062f4d4bb51afd00f6eba991 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:14:27.820075 9750 raft_consensus.cc:482] T 00000000000000000000000000000000 P bace56cc062f4d4bb51afd00f6eba991 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:14:27.820101 9753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:27.780022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52930 (local address 127.29.234.62:33493)
0924 16:14:27.793923 (+ 13901us) server_negotiation.cc:206] Beginning negotiation
0924 16:14:27.793944 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 16:14:27.794089 (+ 145us) server_negotiation.cc:407] Connection header received
0924 16:14:27.797172 (+ 3083us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:14:27.797210 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:14:27.797948 (+ 738us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:14:27.798483 (+ 535us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:14:27.800087 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:27.802295 (+ 2208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:27.804215 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:27.805382 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:27.805584 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:27.810607 (+ 5023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:14:27.810621 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:14:27.810644 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:14:27.810693 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:14:27.814875 (+ 4182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:27.815752 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:27.815760 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:27.815771 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:27.816003 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:27.816460 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:27.816465 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:27.816468 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:27.817364 (+ 896us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:14:27.817663 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0924 16:14:27.818159 (+ 496us) server_negotiation.cc:299] Negotiation successful
0924 16:14:27.818618 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13635,"thread_start_us":13388,"threads_started":1}
I20260924 16:14:27.820299 9750 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bace56cc062f4d4bb51afd00f6eba991 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:14:27.823730 9750 raft_consensus.cc:504] T 00000000000000000000000000000000 P bace56cc062f4d4bb51afd00f6eba991 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bace56cc062f4d4bb51afd00f6eba991" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 33493 } }
I20260924 16:14:27.824316 9750 leader_election.cc:302] T 00000000000000000000000000000000 P bace56cc062f4d4bb51afd00f6eba991 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bace56cc062f4d4bb51afd00f6eba991; no voters:
I20260924 16:14:27.824944 9750 leader_election.cc:288] T 00000000000000000000000000000000 P bace56cc062f4d4bb51afd00f6eba991 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:14:27.825086 9755 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bace56cc062f4d4bb51afd00f6eba991 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:14:27.826962 9750 sys_catalog.cc:564] T 00000000000000000000000000000000 P bace56cc062f4d4bb51afd00f6eba991 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:14:27.835978 9755 raft_consensus.cc:686] T 00000000000000000000000000000000 P bace56cc062f4d4bb51afd00f6eba991 [term 1 LEADER]: Becoming Leader. State: Replica: bace56cc062f4d4bb51afd00f6eba991, State: Running, Role: LEADER
I20260924 16:14:27.837046 9755 consensus_queue.cc:237] T 00000000000000000000000000000000 P bace56cc062f4d4bb51afd00f6eba991 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bace56cc062f4d4bb51afd00f6eba991" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 33493 } }
I20260924 16:14:27.848599 9757 sys_catalog.cc:455] T 00000000000000000000000000000000 P bace56cc062f4d4bb51afd00f6eba991 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bace56cc062f4d4bb51afd00f6eba991. Latest consensus state: current_term: 1 leader_uuid: "bace56cc062f4d4bb51afd00f6eba991" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bace56cc062f4d4bb51afd00f6eba991" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 33493 } } }
I20260924 16:14:27.849041 9757 sys_catalog.cc:458] T 00000000000000000000000000000000 P bace56cc062f4d4bb51afd00f6eba991 [sys.catalog]: This master's current role is: LEADER
I20260924 16:14:27.848465 9756 sys_catalog.cc:455] T 00000000000000000000000000000000 P bace56cc062f4d4bb51afd00f6eba991 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bace56cc062f4d4bb51afd00f6eba991" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bace56cc062f4d4bb51afd00f6eba991" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 33493 } } }
I20260924 16:14:27.849547 9756 sys_catalog.cc:458] T 00000000000000000000000000000000 P bace56cc062f4d4bb51afd00f6eba991 [sys.catalog]: This master's current role is: LEADER
I20260924 16:14:27.850066 9765 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:14:27.869036 9765 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:14:27.896868 9765 catalog_manager.cc:1347] Generated new cluster ID: 6a31aaaa2da541358d2d7a4f64b463b0
I20260924 16:14:27.897037 9765 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:14:27.918780 9765 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:14:27.919330 9750 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 16:14:27.921005 9765 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:14:27.942921 9765 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bace56cc062f4d4bb51afd00f6eba991: Generated new TSK 0
I20260924 16:14:27.944344 9765 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.234.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab.
Entry for principal kudu/127.29.234.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab.
I20260924 16:14:28.770534 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.234.20:45061/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.1:0
--local_ip_for_outbound_sockets=127.29.234.1
--webserver_interface=127.29.234.1
--webserver_port=0
--tserver_master_addrs=127.29.234.62:33493
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab
--principal=kudu/127.29.234.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:44353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 16:14:29 dist-test-slave-v5j9 krb5kdc[9229](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790266469, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 16:14:29.236225 9797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:14:29.236523 9797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:14:29.236682 9797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:14:29.244333 9797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:14:29.244442 9797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:14:29.244663 9797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.1
I20260924 16:14:29.254910 9797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:44353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.29.234.20:45061/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab
--principal=kudu/127.29.234.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.234.1
--webserver_port=0
--tserver_master_addrs=127.29.234.62:33493
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.1
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:14:29.256281 9797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:14:29.257704 9797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:14:29.291189 9808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:14:29.292466 9807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:14:29.304236 9810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:14:29.304860 9797 server_base.cc:1034] running on GCE node
I20260924 16:14:29.305768 9797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:14:29.307089 9797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:14:29.308442 9797 hybrid_clock.cc:648] HybridClock initialized: now 1790266469308310 us; error 98 us; skew 500 ppm
I20260924 16:14:29.308856 9797 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:14:29 dist-test-slave-v5j9 krb5kdc[9229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266469, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:14:29.313154 9797 init.cc:377] Logged in from keytab as kudu/127.29.234.1@KRBTEST.COM (short username kudu)
I20260924 16:14:29.318889 9797 webserver.cc:460] Webserver started at http://127.29.234.1:41931/ using document root <none> and password file <none>
I20260924 16:14:29.319936 9797 fs_manager.cc:359] Metadata directory not provided
I20260924 16:14:29.320077 9797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:14:29.320453 9797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 16:14:29 dist-test-slave-v5j9 krb5kdc[9229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266469, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.1@KRBTEST.COM for HTTP/127.29.234.20@KRBTEST.COM
I20260924 16:14:29.346371 9797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/ts-0/data/instance:
uuid: "a420200281df441fb040a9357a7cda2f"
format_stamp: "Formatted at 2026-09-24 16:14:29 on dist-test-slave-v5j9"
server_key: "258fcf430dcad6f70e6ee77ba754f013"
server_key_iv: "d4abfe860d6ae8e0da3708263071c807"
server_key_version: "kuduclusterkey@0"
I20260924 16:14:29.347638 9797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/ts-0/wal/instance:
uuid: "a420200281df441fb040a9357a7cda2f"
format_stamp: "Formatted at 2026-09-24 16:14:29 on dist-test-slave-v5j9"
server_key: "258fcf430dcad6f70e6ee77ba754f013"
server_key_iv: "d4abfe860d6ae8e0da3708263071c807"
server_key_version: "kuduclusterkey@0"
I20260924 16:14:29.361672 9797 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.001s sys 0.012s
I20260924 16:14:29.391093 9817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:14:29.393273 9797 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.002s sys 0.007s
I20260924 16:14:29.393604 9797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/ts-0/wal
uuid: "a420200281df441fb040a9357a7cda2f"
format_stamp: "Formatted at 2026-09-24 16:14:29 on dist-test-slave-v5j9"
server_key: "258fcf430dcad6f70e6ee77ba754f013"
server_key_iv: "d4abfe860d6ae8e0da3708263071c807"
server_key_version: "kuduclusterkey@0"
I20260924 16:14:29.393983 9797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:14:29.431911 9797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:14:29.440953 9797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:14:29.441416 9797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:14:29.465651 9797 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
Sep 24 16:14:29 dist-test-slave-v5j9 krb5kdc[9229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266469, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.62@KRBTEST.COM for HTTP/127.29.234.20@KRBTEST.COM
I20260924 16:14:29.482189 9797 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:14:29.482307 9797 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:14:29.482441 9797 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:14:29.482498 9797 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 16:14:29 dist-test-slave-v5j9 krb5kdc[9229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266469, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.62@KRBTEST.COM for HTTP/127.29.234.20@KRBTEST.COM
I20260924 16:14:29.841903 9931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.1:32953 every 8 connection(s)
I20260924 16:14:29.842754 9797 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.1:32953
I20260924 16:14:29.847023 9797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
I20260924 16:14:29.856165 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 9797
I20260924 16:14:29.856411 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/ts-0/wal/instance
I20260924 16:14:29.857299 30632 external_mini_cluster.cc:1428] Setting key 0fa5e56927e0fcdd2444cd518d7eda39
WARNING: no policy specified for kudu/127.29.234.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.2@KRBTEST.COM" created.
Sep 24 16:14:29 dist-test-slave-v5j9 krb5kdc[9229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266469, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.1@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:14:29.923173 9934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:29.856831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.1:33631 (local address 127.29.234.62:33493)
0924 16:14:29.881829 (+ 24998us) server_negotiation.cc:206] Beginning negotiation
0924 16:14:29.881834 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:14:29.881853 (+ 19us) server_negotiation.cc:407] Connection header received
0924 16:14:29.884088 (+ 2235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:14:29.884098 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:14:29.884239 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:14:29.884354 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:14:29.890441 (+ 6087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:29.891550 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:29.894874 (+ 3324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:29.896119 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:29.896268 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:29.905275 (+ 9007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:14:29.905284 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:14:29.905289 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:14:29.905337 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:14:29.914439 (+ 9102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:29.918202 (+ 3763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:29.918211 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:29.918218 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:29.918389 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:29.919033 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:29.919039 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:29.919042 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:29.919480 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:14:29.919632 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 16:14:29.922355 (+ 2723us) server_negotiation.cc:299] Negotiation successful
0924 16:14:29.922645 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24852,"thread_start_us":24672,"threads_started":1}
I20260924 16:14:29.929641 9935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:29.857597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:33493 (local address 127.29.234.1:33631)
0924 16:14:29.874965 (+ 17368us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:29.875045 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0924 16:14:29.882607 (+ 7562us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:14:29.884911 (+ 2304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:14:29.884926 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:14:29.889097 (+ 4171us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:14:29.890145 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:29.890179 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:29.891777 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:29.891782 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:29.894685 (+ 2903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:29.894702 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:29.897318 (+ 2616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:29.897330 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:29.898819 (+ 1489us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:29.900191 (+ 1372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:14:29.900233 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:14:29.904734 (+ 4501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:14:29.915402 (+ 10668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:14:29.915430 (+ 28us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:14:29.915455 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:14:29.918029 (+ 2574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:14:29.918541 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:14:29.918549 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:14:29.918555 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:14:29.918874 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:14:29.919810 (+ 936us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:14:29.919820 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:14:29.920927 (+ 1107us) client_negotiation.cc:769] Sending connection context
0924 16:14:29.921459 (+ 532us) client_negotiation.cc:240] Negotiation successful
0924 16:14:29.925161 (+ 3702us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16929,"thread_start_us":19649,"threads_started":1}
I20260924 16:14:29.933045 9932 heartbeater.cc:347] Connected to a master server at 127.29.234.62:33493
I20260924 16:14:29.933480 9932 heartbeater.cc:464] Registering TS with master...
I20260924 16:14:29.934526 9932 heartbeater.cc:511] Master 127.29.234.62:33493 requested a full tablet report, sending...
I20260924 16:14:29.937461 9715 ts_manager.cc:194] Registered new tserver with Master: a420200281df441fb040a9357a7cda2f (127.29.234.1:32953)
I20260924 16:14:29.939581 9715 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.234.1@KRBTEST.COM'} at 127.29.234.1:33631
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab.
Entry for principal kudu/127.29.234.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab.
W20260924 16:14:29.963725 30632 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.29.234.2: real 0.103s user 0.000s sys 0.018s
I20260924 16:14:29.964681 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.234.20:45061/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.2:0
--local_ip_for_outbound_sockets=127.29.234.2
--webserver_interface=127.29.234.2
--webserver_port=0
--tserver_master_addrs=127.29.234.62:33493
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab
--principal=kudu/127.29.234.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:44353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:14:30.368777 9939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:14:30.369099 9939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:14:30.369266 9939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:14:30.377110 9939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:14:30.377267 9939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:14:30.377476 9939 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.2
I20260924 16:14:30.387410 9939 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:44353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.29.234.20:45061/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab
--principal=kudu/127.29.234.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.234.2
--webserver_port=0
--tserver_master_addrs=127.29.234.62:33493
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.2
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:14:30.388803 9939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:14:30.390230 9939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:14:30.405503 9951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:14:30.405483 9952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:14:30.406400 9954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:14:30.406956 9939 server_base.cc:1034] running on GCE node
I20260924 16:14:30.407740 9939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:14:30.408777 9939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:14:30.410018 9939 hybrid_clock.cc:648] HybridClock initialized: now 1790266470409951 us; error 51 us; skew 500 ppm
I20260924 16:14:30.410380 9939 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:14:30 dist-test-slave-v5j9 krb5kdc[9229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266470, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:14:30.413723 9939 init.cc:377] Logged in from keytab as kudu/127.29.234.2@KRBTEST.COM (short username kudu)
I20260924 16:14:30.415373 9939 webserver.cc:460] Webserver started at http://127.29.234.2:43607/ using document root <none> and password file <none>
I20260924 16:14:30.416162 9939 fs_manager.cc:359] Metadata directory not provided
I20260924 16:14:30.416302 9939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:14:30.416633 9939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 16:14:30 dist-test-slave-v5j9 krb5kdc[9229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266470, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.2@KRBTEST.COM for HTTP/127.29.234.20@KRBTEST.COM
I20260924 16:14:30.435276 9939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/ts-1/data/instance:
uuid: "d6982669590a45e6a7e4aed0e44b67e6"
format_stamp: "Formatted at 2026-09-24 16:14:30 on dist-test-slave-v5j9"
server_key: "c49047427596070bbd37d1fb2bbdd340"
server_key_iv: "449884070c9fc7b0f60bf4888aa6b5c9"
server_key_version: "kuduclusterkey@0"
I20260924 16:14:30.436439 9939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/ts-1/wal/instance:
uuid: "d6982669590a45e6a7e4aed0e44b67e6"
format_stamp: "Formatted at 2026-09-24 16:14:30 on dist-test-slave-v5j9"
server_key: "c49047427596070bbd37d1fb2bbdd340"
server_key_iv: "449884070c9fc7b0f60bf4888aa6b5c9"
server_key_version: "kuduclusterkey@0"
I20260924 16:14:30.443557 9939 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.003s
I20260924 16:14:30.462226 9961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:14:30.464257 9939 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 16:14:30.464527 9939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/ts-1/wal
uuid: "d6982669590a45e6a7e4aed0e44b67e6"
format_stamp: "Formatted at 2026-09-24 16:14:30 on dist-test-slave-v5j9"
server_key: "c49047427596070bbd37d1fb2bbdd340"
server_key_iv: "449884070c9fc7b0f60bf4888aa6b5c9"
server_key_version: "kuduclusterkey@0"
I20260924 16:14:30.464898 9939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:14:30.493431 9939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:14:30.498863 9939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:14:30.499274 9939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:14:30.500955 9939 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:14:30.503998 9939 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:14:30.504158 9939 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260924 16:14:30.504324 9939 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:14:30.504422 9939 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:14:30.550518 9939 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.2:46021
I20260924 16:14:30.550560 10076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.2:46021 every 8 connection(s)
I20260924 16:14:30.552464 9939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
I20260924 16:14:30.554800 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 9939
I20260924 16:14:30.555004 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/ts-1/wal/instance
I20260924 16:14:30.555536 30632 external_mini_cluster.cc:1428] Setting key eeba6d685fbc2d21971dfbd10197f96a
WARNING: no policy specified for kudu/127.29.234.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.3@KRBTEST.COM" created.
Sep 24 16:14:30 dist-test-slave-v5j9 krb5kdc[9229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266470, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.2@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:14:30.604768 10079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:30.557712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.2:35855 (local address 127.29.234.62:33493)
0924 16:14:30.578546 (+ 20834us) server_negotiation.cc:206] Beginning negotiation
0924 16:14:30.578552 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:14:30.578575 (+ 23us) server_negotiation.cc:407] Connection header received
0924 16:14:30.578643 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:14:30.578648 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:14:30.578746 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:14:30.578864 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:14:30.581205 (+ 2341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:30.582521 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:30.585290 (+ 2769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:30.586120 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:30.586326 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:30.593919 (+ 7593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:14:30.593930 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:14:30.593938 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:14:30.593998 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:14:30.599034 (+ 5036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:30.600286 (+ 1252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:30.600295 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:30.600301 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:30.600457 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:30.601101 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:30.601109 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:30.601114 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:30.601577 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:14:30.601718 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 16:14:30.603951 (+ 2233us) server_negotiation.cc:299] Negotiation successful
0924 16:14:30.604206 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20686,"thread_start_us":20560,"threads_started":1}
I20260924 16:14:30.606422 10080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:30.558662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:33493 (local address 127.29.234.2:35855)
0924 16:14:30.572413 (+ 13751us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:30.572472 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 16:14:30.574920 (+ 2448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:14:30.579235 (+ 4315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:14:30.579251 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:14:30.579997 (+ 746us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:14:30.580955 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:30.580985 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:30.582722 (+ 1737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:30.582730 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:30.585075 (+ 2345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:30.585098 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:30.586683 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:30.586691 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:30.587050 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:30.587932 (+ 882us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:14:30.587975 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:14:30.593568 (+ 5593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:14:30.599265 (+ 5697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:14:30.599278 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:14:30.599303 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:14:30.600131 (+ 828us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:14:30.600591 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:14:30.600599 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:14:30.600605 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:14:30.600961 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:14:30.601818 (+ 857us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:14:30.601829 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:14:30.602328 (+ 499us) client_negotiation.cc:769] Sending connection context
0924 16:14:30.602682 (+ 354us) client_negotiation.cc:240] Negotiation successful
0924 16:14:30.603295 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13397,"thread_start_us":13089,"threads_started":1}
I20260924 16:14:30.608592 10077 heartbeater.cc:347] Connected to a master server at 127.29.234.62:33493
I20260924 16:14:30.608969 10077 heartbeater.cc:464] Registering TS with master...
I20260924 16:14:30.609997 10077 heartbeater.cc:511] Master 127.29.234.62:33493 requested a full tablet report, sending...
I20260924 16:14:30.611809 9715 ts_manager.cc:194] Registered new tserver with Master: d6982669590a45e6a7e4aed0e44b67e6 (127.29.234.2:46021)
I20260924 16:14:30.613133 9715 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.234.2@KRBTEST.COM'} at 127.29.234.2:35855
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab.
Entry for principal kudu/127.29.234.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab.
I20260924 16:14:30.628706 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.234.20:45061/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.3:0
--local_ip_for_outbound_sockets=127.29.234.3
--webserver_interface=127.29.234.3
--webserver_port=0
--tserver_master_addrs=127.29.234.62:33493
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab
--principal=kudu/127.29.234.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:44353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:14:30.943774 9932 heartbeater.cc:503] Master 127.29.234.62:33493 was elected leader, sending a full tablet report...
W20260924 16:14:31.072275 10083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:14:31.072610 10083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:14:31.072774 10083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:14:31.080864 10083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:14:31.081043 10083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:14:31.081259 10083 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.3
I20260924 16:14:31.091631 10083 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:44353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.29.234.20:45061/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab
--principal=kudu/127.29.234.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.234.3
--webserver_port=0
--tserver_master_addrs=127.29.234.62:33493
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.3
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:14:31.093055 10083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:14:31.094568 10083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:14:31.109700 10088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:14:31.109779 10091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:14:31.109754 10089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:14:31.110351 10083 server_base.cc:1034] running on GCE node
I20260924 16:14:31.111126 10083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:14:31.112197 10083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:14:31.113462 10083 hybrid_clock.cc:648] HybridClock initialized: now 1790266471113395 us; error 55 us; skew 500 ppm
I20260924 16:14:31.113807 10083 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:14:31 dist-test-slave-v5j9 krb5kdc[9229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266471, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:14:31.116991 10083 init.cc:377] Logged in from keytab as kudu/127.29.234.3@KRBTEST.COM (short username kudu)
I20260924 16:14:31.118721 10083 webserver.cc:460] Webserver started at http://127.29.234.3:44383/ using document root <none> and password file <none>
I20260924 16:14:31.119573 10083 fs_manager.cc:359] Metadata directory not provided
I20260924 16:14:31.119740 10083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:14:31.120107 10083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 16:14:31 dist-test-slave-v5j9 krb5kdc[9229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266471, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.3@KRBTEST.COM for HTTP/127.29.234.20@KRBTEST.COM
I20260924 16:14:31.137125 10083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/ts-2/data/instance:
uuid: "5db508082b8e4c2b84d6026dc4e167ba"
format_stamp: "Formatted at 2026-09-24 16:14:31 on dist-test-slave-v5j9"
server_key: "68a503db01453ef688539fd1b415a7b0"
server_key_iv: "2b3a6e026fc21f3a229495ffb129800b"
server_key_version: "kuduclusterkey@0"
I20260924 16:14:31.138190 10083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/ts-2/wal/instance:
uuid: "5db508082b8e4c2b84d6026dc4e167ba"
format_stamp: "Formatted at 2026-09-24 16:14:31 on dist-test-slave-v5j9"
server_key: "68a503db01453ef688539fd1b415a7b0"
server_key_iv: "2b3a6e026fc21f3a229495ffb129800b"
server_key_version: "kuduclusterkey@0"
I20260924 16:14:31.145032 10083 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260924 16:14:31.161599 10098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:14:31.163242 10083 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 16:14:31.163437 10083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/ts-2/wal
uuid: "5db508082b8e4c2b84d6026dc4e167ba"
format_stamp: "Formatted at 2026-09-24 16:14:31 on dist-test-slave-v5j9"
server_key: "68a503db01453ef688539fd1b415a7b0"
server_key_iv: "2b3a6e026fc21f3a229495ffb129800b"
server_key_version: "kuduclusterkey@0"
I20260924 16:14:31.163679 10083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:14:31.185070 10083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:14:31.188824 10083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:14:31.189136 10083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:14:31.190203 10083 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:14:31.191552 10083 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:14:31.191653 10083 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:14:31.191738 10083 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:14:31.191782 10083 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:14:31.225201 10083 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.3:36979
I20260924 16:14:31.225258 10213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.3:36979 every 8 connection(s)
I20260924 16:14:31.226605 10083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
I20260924 16:14:31.230353 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 10083
I20260924 16:14:31.230556 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/ts-2/wal/instance
I20260924 16:14:31.231163 30632 external_mini_cluster.cc:1428] Setting key 428f29f12b6f14dca279b5fb9e3f8d9a
Sep 24 16:14:31 dist-test-slave-v5j9 krb5kdc[9229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266471, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.3@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:14:31.247279 10216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:31.229282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.3:46627 (local address 127.29.234.62:33493)
0924 16:14:31.229788 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0924 16:14:31.229793 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:14:31.230967 (+ 1174us) server_negotiation.cc:407] Connection header received
0924 16:14:31.233411 (+ 2444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:14:31.233416 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:14:31.233549 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:14:31.233660 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:14:31.235298 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:31.236033 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:31.237605 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:31.238051 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:31.238143 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:31.241413 (+ 3270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:14:31.241424 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:14:31.241427 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:14:31.241459 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:14:31.243889 (+ 2430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:31.244533 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:31.244537 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:31.244539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:31.244655 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:31.245216 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:31.245222 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:31.245225 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:31.245677 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:14:31.245844 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 16:14:31.246591 (+ 747us) server_negotiation.cc:299] Negotiation successful
0924 16:14:31.246824 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":301,"threads_started":1}
I20260924 16:14:31.248303 10217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:31.229858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:33493 (local address 127.29.234.3:46627)
0924 16:14:31.230764 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:31.230829 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0924 16:14:31.233060 (+ 2231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:14:31.233876 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:14:31.233885 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:14:31.234477 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:14:31.235143 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:31.235161 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:31.236161 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:31.236164 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:31.237479 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:31.237491 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:31.238239 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:31.238243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:31.238447 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:31.239138 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:14:31.239165 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:14:31.241167 (+ 2002us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:14:31.244012 (+ 2845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:14:31.244019 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:14:31.244032 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:14:31.244437 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:14:31.244771 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:14:31.244775 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:14:31.244778 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:14:31.245037 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:14:31.245806 (+ 769us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:14:31.245814 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:14:31.246162 (+ 348us) client_negotiation.cc:769] Sending connection context
0924 16:14:31.246515 (+ 353us) client_negotiation.cc:240] Negotiation successful
0924 16:14:31.247065 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":318,"threads_started":1}
I20260924 16:14:31.249688 10214 heartbeater.cc:347] Connected to a master server at 127.29.234.62:33493
I20260924 16:14:31.250029 10214 heartbeater.cc:464] Registering TS with master...
I20260924 16:14:31.250779 10214 heartbeater.cc:511] Master 127.29.234.62:33493 requested a full tablet report, sending...
I20260924 16:14:31.252287 9715 ts_manager.cc:194] Registered new tserver with Master: 5db508082b8e4c2b84d6026dc4e167ba (127.29.234.3:36979)
I20260924 16:14:31.253057 9715 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.234.3@KRBTEST.COM'} at 127.29.234.3:46627
I20260924 16:14:31.259326 30632 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:14:31.277192 10216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:31.264119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52946 (local address 127.29.234.62:33493)
0924 16:14:31.264417 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0924 16:14:31.264424 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:14:31.264702 (+ 278us) server_negotiation.cc:407] Connection header received
0924 16:14:31.264914 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:14:31.264921 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:14:31.265017 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:14:31.265126 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:14:31.266712 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:31.267540 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:31.268918 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:31.269407 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:31.269544 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:31.271987 (+ 2443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:14:31.271997 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:14:31.272001 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:14:31.272031 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:14:31.274541 (+ 2510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:31.275218 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:31.275224 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:31.275227 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:31.275370 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:31.275758 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:31.275765 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:31.275769 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:31.276033 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:14:31.276306 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0924 16:14:31.276613 (+ 307us) server_negotiation.cc:299] Negotiation successful
0924 16:14:31.276841 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260924 16:14:31.281800 9715 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52946:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:14:31.292282 9715 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:14:31.329777 10229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:31.309638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55556 (local address 127.29.234.2:46021)
0924 16:14:31.310789 (+ 1151us) server_negotiation.cc:206] Beginning negotiation
0924 16:14:31.310801 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 16:14:31.319904 (+ 9103us) server_negotiation.cc:407] Connection header received
0924 16:14:31.320000 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:14:31.320006 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:14:31.320235 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:14:31.320440 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:14:31.321953 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:31.323602 (+ 1649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:31.326856 (+ 3254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:31.328250 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:31.328392 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:31.328662 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0924 16:14:31.329090 (+ 428us) server_negotiation.cc:299] Negotiation successful
0924 16:14:31.329285 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":913,"thread_start_us":697,"threads_started":1}
I20260924 16:14:31.330091 10231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:31.313126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.2:46021 (local address 127.0.0.1:55556)
0924 16:14:31.319628 (+ 6502us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:31.319654 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 16:14:31.319829 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:14:31.320560 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:14:31.320567 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:14:31.320593 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:14:31.321756 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:31.321769 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:31.323774 (+ 2005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:31.323779 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:31.326590 (+ 2811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:31.326604 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:31.328559 (+ 1955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:31.328566 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:31.328844 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:31.328886 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 16:14:31.329019 (+ 133us) client_negotiation.cc:240] Negotiation successful
0924 16:14:31.329274 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5653,"mutex_wait_us":4,"spinlock_wait_cycles":3968,"thread_start_us":4375,"threads_started":1}
I20260924 16:14:31.332957 10227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:31.313335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.3:36979 (local address 127.0.0.1:37464)
0924 16:14:31.319666 (+ 6331us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:31.319684 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 16:14:31.319794 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:14:31.320605 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:14:31.320611 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:14:31.320635 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:14:31.321606 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:31.321681 (+ 75us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:31.323780 (+ 2099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:31.323786 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:31.327671 (+ 3885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:31.327697 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:31.331514 (+ 3817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:31.331525 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:31.331883 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:31.331944 (+ 61us) client_negotiation.cc:769] Sending connection context
0924 16:14:31.332039 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 16:14:31.332198 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6141,"mutex_wait_us":63,"thread_start_us":5224,"threads_started":1}
I20260924 16:14:31.333030 10232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:31.314201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37464 (local address 127.29.234.3:36979)
0924 16:14:31.319456 (+ 5255us) server_negotiation.cc:206] Beginning negotiation
0924 16:14:31.319468 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 16:14:31.319794 (+ 326us) server_negotiation.cc:407] Connection header received
0924 16:14:31.319978 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:14:31.319986 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:14:31.320236 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:14:31.320440 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:14:31.321932 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:31.323585 (+ 1653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:31.327892 (+ 4307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:31.330753 (+ 2861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:31.330882 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:31.331231 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0924 16:14:31.332145 (+ 914us) server_negotiation.cc:299] Negotiation successful
0924 16:14:31.332371 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5051,"thread_start_us":4838,"threads_started":1}
I20260924 16:14:31.333300 10228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:31.309064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48576 (local address 127.29.234.1:32953)
0924 16:14:31.310058 (+ 994us) server_negotiation.cc:206] Beginning negotiation
0924 16:14:31.310067 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:14:31.318322 (+ 8255us) server_negotiation.cc:407] Connection header received
0924 16:14:31.319835 (+ 1513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:14:31.319844 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:14:31.320107 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:14:31.320315 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:14:31.321684 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:31.323210 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:31.328671 (+ 5461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:31.330991 (+ 2320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:31.331082 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:31.331477 (+ 395us) server_negotiation.cc:1035] Waiting for connection context
0924 16:14:31.332374 (+ 897us) server_negotiation.cc:299] Negotiation successful
0924 16:14:31.332630 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":768,"thread_start_us":835,"threads_started":1}
I20260924 16:14:31.334836 10230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:31.308900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:32953 (local address 127.0.0.1:48576)
0924 16:14:31.318088 (+ 9188us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:31.318144 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 16:14:31.319479 (+ 1335us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:14:31.320315 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:14:31.320323 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:14:31.320360 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:14:31.321450 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:31.321470 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:31.323335 (+ 1865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:31.323342 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:31.328411 (+ 5069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:31.328443 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:31.331885 (+ 3442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:31.331892 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:31.332158 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:31.332188 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 16:14:31.333590 (+ 1402us) client_negotiation.cc:240] Negotiation successful
0924 16:14:31.333778 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8792,"mutex_wait_us":43,"spinlock_wait_cycles":3968,"thread_start_us":10456,"threads_started":1}
I20260924 16:14:31.334549 10010 tablet_service.cc:1467] Processing CreateTablet for tablet 7884aee96af1427099297ef39e1d65e3 (DEFAULT_TABLE table=test-table [id=1f392921510e45c8b6e9c635802dd690]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:14:31.336254 10010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7884aee96af1427099297ef39e1d65e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:14:31.338022 10147 tablet_service.cc:1467] Processing CreateTablet for tablet 7884aee96af1427099297ef39e1d65e3 (DEFAULT_TABLE table=test-table [id=1f392921510e45c8b6e9c635802dd690]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:14:31.339888 10147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7884aee96af1427099297ef39e1d65e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:14:31.339507 9866 tablet_service.cc:1467] Processing CreateTablet for tablet 7884aee96af1427099297ef39e1d65e3 (DEFAULT_TABLE table=test-table [id=1f392921510e45c8b6e9c635802dd690]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:14:31.341763 9866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7884aee96af1427099297ef39e1d65e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:14:31.354321 10233 tablet_bootstrap.cc:492] T 7884aee96af1427099297ef39e1d65e3 P 5db508082b8e4c2b84d6026dc4e167ba: Bootstrap starting.
I20260924 16:14:31.358230 10233 tablet_bootstrap.cc:654] T 7884aee96af1427099297ef39e1d65e3 P 5db508082b8e4c2b84d6026dc4e167ba: Neither blocks nor log segments found. Creating new log.
I20260924 16:14:31.359562 10233 log.cc:824] T 7884aee96af1427099297ef39e1d65e3 P 5db508082b8e4c2b84d6026dc4e167ba: Log is configured to *not* fsync() on all Append() calls
I20260924 16:14:31.360595 10234 tablet_bootstrap.cc:492] T 7884aee96af1427099297ef39e1d65e3 P d6982669590a45e6a7e4aed0e44b67e6: Bootstrap starting.
I20260924 16:14:31.361084 10235 tablet_bootstrap.cc:492] T 7884aee96af1427099297ef39e1d65e3 P a420200281df441fb040a9357a7cda2f: Bootstrap starting.
I20260924 16:14:31.363144 10233 tablet_bootstrap.cc:492] T 7884aee96af1427099297ef39e1d65e3 P 5db508082b8e4c2b84d6026dc4e167ba: No bootstrap required, opened a new log
I20260924 16:14:31.363427 10233 ts_tablet_manager.cc:1397] T 7884aee96af1427099297ef39e1d65e3 P 5db508082b8e4c2b84d6026dc4e167ba: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260924 16:14:31.364661 10234 tablet_bootstrap.cc:654] T 7884aee96af1427099297ef39e1d65e3 P d6982669590a45e6a7e4aed0e44b67e6: Neither blocks nor log segments found. Creating new log.
I20260924 16:14:31.365834 10234 log.cc:824] T 7884aee96af1427099297ef39e1d65e3 P d6982669590a45e6a7e4aed0e44b67e6: Log is configured to *not* fsync() on all Append() calls
I20260924 16:14:31.368721 10235 tablet_bootstrap.cc:654] T 7884aee96af1427099297ef39e1d65e3 P a420200281df441fb040a9357a7cda2f: Neither blocks nor log segments found. Creating new log.
I20260924 16:14:31.369571 10234 tablet_bootstrap.cc:492] T 7884aee96af1427099297ef39e1d65e3 P d6982669590a45e6a7e4aed0e44b67e6: No bootstrap required, opened a new log
I20260924 16:14:31.369863 10234 ts_tablet_manager.cc:1397] T 7884aee96af1427099297ef39e1d65e3 P d6982669590a45e6a7e4aed0e44b67e6: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.005s
I20260924 16:14:31.370054 10235 log.cc:824] T 7884aee96af1427099297ef39e1d65e3 P a420200281df441fb040a9357a7cda2f: Log is configured to *not* fsync() on all Append() calls
I20260924 16:14:31.372910 10235 tablet_bootstrap.cc:492] T 7884aee96af1427099297ef39e1d65e3 P a420200281df441fb040a9357a7cda2f: No bootstrap required, opened a new log
I20260924 16:14:31.373184 10235 ts_tablet_manager.cc:1397] T 7884aee96af1427099297ef39e1d65e3 P a420200281df441fb040a9357a7cda2f: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20260924 16:14:31.373260 10233 raft_consensus.cc:348] T 7884aee96af1427099297ef39e1d65e3 P 5db508082b8e4c2b84d6026dc4e167ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a420200281df441fb040a9357a7cda2f" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 32953 } } peers { permanent_uuid: "d6982669590a45e6a7e4aed0e44b67e6" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 46021 } } peers { permanent_uuid: "5db508082b8e4c2b84d6026dc4e167ba" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 36979 } }
I20260924 16:14:31.373756 10233 raft_consensus.cc:374] T 7884aee96af1427099297ef39e1d65e3 P 5db508082b8e4c2b84d6026dc4e167ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:14:31.373867 10233 raft_consensus.cc:729] T 7884aee96af1427099297ef39e1d65e3 P 5db508082b8e4c2b84d6026dc4e167ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5db508082b8e4c2b84d6026dc4e167ba, State: Initialized, Role: FOLLOWER
I20260924 16:14:31.374526 10233 consensus_queue.cc:260] T 7884aee96af1427099297ef39e1d65e3 P 5db508082b8e4c2b84d6026dc4e167ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a420200281df441fb040a9357a7cda2f" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 32953 } } peers { permanent_uuid: "d6982669590a45e6a7e4aed0e44b67e6" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 46021 } } peers { permanent_uuid: "5db508082b8e4c2b84d6026dc4e167ba" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 36979 } }
I20260924 16:14:31.375690 10214 heartbeater.cc:503] Master 127.29.234.62:33493 was elected leader, sending a full tablet report...
I20260924 16:14:31.376369 10233 ts_tablet_manager.cc:1428] T 7884aee96af1427099297ef39e1d65e3 P 5db508082b8e4c2b84d6026dc4e167ba: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
I20260924 16:14:31.379454 10234 raft_consensus.cc:348] T 7884aee96af1427099297ef39e1d65e3 P d6982669590a45e6a7e4aed0e44b67e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a420200281df441fb040a9357a7cda2f" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 32953 } } peers { permanent_uuid: "d6982669590a45e6a7e4aed0e44b67e6" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 46021 } } peers { permanent_uuid: "5db508082b8e4c2b84d6026dc4e167ba" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 36979 } }
I20260924 16:14:31.379843 10234 raft_consensus.cc:374] T 7884aee96af1427099297ef39e1d65e3 P d6982669590a45e6a7e4aed0e44b67e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:14:31.379969 10234 raft_consensus.cc:729] T 7884aee96af1427099297ef39e1d65e3 P d6982669590a45e6a7e4aed0e44b67e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6982669590a45e6a7e4aed0e44b67e6, State: Initialized, Role: FOLLOWER
I20260924 16:14:31.380597 10234 consensus_queue.cc:260] T 7884aee96af1427099297ef39e1d65e3 P d6982669590a45e6a7e4aed0e44b67e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a420200281df441fb040a9357a7cda2f" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 32953 } } peers { permanent_uuid: "d6982669590a45e6a7e4aed0e44b67e6" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 46021 } } peers { permanent_uuid: "5db508082b8e4c2b84d6026dc4e167ba" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 36979 } }
I20260924 16:14:31.381901 10077 heartbeater.cc:503] Master 127.29.234.62:33493 was elected leader, sending a full tablet report...
I20260924 16:14:31.382634 10234 ts_tablet_manager.cc:1428] T 7884aee96af1427099297ef39e1d65e3 P d6982669590a45e6a7e4aed0e44b67e6: Time spent starting tablet: real 0.013s user 0.009s sys 0.003s
I20260924 16:14:31.383589 10235 raft_consensus.cc:348] T 7884aee96af1427099297ef39e1d65e3 P a420200281df441fb040a9357a7cda2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a420200281df441fb040a9357a7cda2f" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 32953 } } peers { permanent_uuid: "d6982669590a45e6a7e4aed0e44b67e6" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 46021 } } peers { permanent_uuid: "5db508082b8e4c2b84d6026dc4e167ba" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 36979 } }
I20260924 16:14:31.384083 10235 raft_consensus.cc:374] T 7884aee96af1427099297ef39e1d65e3 P a420200281df441fb040a9357a7cda2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:14:31.384215 10235 raft_consensus.cc:729] T 7884aee96af1427099297ef39e1d65e3 P a420200281df441fb040a9357a7cda2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a420200281df441fb040a9357a7cda2f, State: Initialized, Role: FOLLOWER
I20260924 16:14:31.384863 10235 consensus_queue.cc:260] T 7884aee96af1427099297ef39e1d65e3 P a420200281df441fb040a9357a7cda2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a420200281df441fb040a9357a7cda2f" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 32953 } } peers { permanent_uuid: "d6982669590a45e6a7e4aed0e44b67e6" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 46021 } } peers { permanent_uuid: "5db508082b8e4c2b84d6026dc4e167ba" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 36979 } }
I20260924 16:14:31.386947 10235 ts_tablet_manager.cc:1428] T 7884aee96af1427099297ef39e1d65e3 P a420200281df441fb040a9357a7cda2f: Time spent starting tablet: real 0.014s user 0.011s sys 0.003s
I20260924 16:14:31.427397 10241 raft_consensus.cc:482] T 7884aee96af1427099297ef39e1d65e3 P a420200281df441fb040a9357a7cda2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:14:31.427737 10241 raft_consensus.cc:504] T 7884aee96af1427099297ef39e1d65e3 P a420200281df441fb040a9357a7cda2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a420200281df441fb040a9357a7cda2f" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 32953 } } peers { permanent_uuid: "d6982669590a45e6a7e4aed0e44b67e6" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 46021 } } peers { permanent_uuid: "5db508082b8e4c2b84d6026dc4e167ba" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 36979 } }
I20260924 16:14:31.429672 10241 leader_election.cc:288] T 7884aee96af1427099297ef39e1d65e3 P a420200281df441fb040a9357a7cda2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6982669590a45e6a7e4aed0e44b67e6 (127.29.234.2:46021), 5db508082b8e4c2b84d6026dc4e167ba (127.29.234.3:36979)
I20260924 16:14:31.436961 10229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:31.429973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.1:40477 (local address 127.29.234.2:46021)
0924 16:14:31.430193 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0924 16:14:31.430199 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:14:31.431036 (+ 837us) server_negotiation.cc:407] Connection header received
0924 16:14:31.431318 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:14:31.431323 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:14:31.431385 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:14:31.431451 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:14:31.432106 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:31.432814 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:31.435183 (+ 2369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:31.435927 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:31.436036 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:31.436121 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 16:14:31.436563 (+ 442us) server_negotiation.cc:299] Negotiation successful
0924 16:14:31.436689 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 16:14:31.436965 10232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:31.430296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.1:53661 (local address 127.29.234.3:36979)
0924 16:14:31.430553 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0924 16:14:31.430559 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:14:31.431163 (+ 604us) server_negotiation.cc:407] Connection header received
0924 16:14:31.431334 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:14:31.431338 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:14:31.431396 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:14:31.431510 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:14:31.432181 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:31.432918 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:31.435234 (+ 2316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:31.436010 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:31.436162 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:31.436237 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 16:14:31.436598 (+ 361us) server_negotiation.cc:299] Negotiation successful
0924 16:14:31.436718 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260924 16:14:31.437103 10242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:31.430050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.2:46021 (local address 127.29.234.1:40477)
0924 16:14:31.430931 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:31.430956 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 16:14:31.431127 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:14:31.431508 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:14:31.431513 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:14:31.431534 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:14:31.431966 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:31.431975 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:31.432949 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:31.432957 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:31.435036 (+ 2079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:31.435048 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:31.436137 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:31.436146 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:31.436333 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:31.436434 (+ 101us) client_negotiation.cc:769] Sending connection context
0924 16:14:31.436508 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 16:14:31.436654 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":724,"mutex_wait_us":57,"thread_start_us":451,"threads_started":1}
I20260924 16:14:31.437609 10243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:31.430156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.3:36979 (local address 127.29.234.1:53661)
0924 16:14:31.431053 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0924 16:14:31.431067 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:14:31.431170 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:14:31.431544 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:14:31.431548 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:14:31.431570 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:14:31.431998 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:31.432006 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:31.433062 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:31.433065 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:31.435056 (+ 1991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:14:31.435068 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:14:31.436238 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:14:31.436242 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:14:31.436380 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:31.436453 (+ 73us) client_negotiation.cc:769] Sending connection context
0924 16:14:31.436557 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 16:14:31.436679 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":808,"mutex_wait_us":46,"thread_start_us":622,"threads_started":1}
I20260924 16:14:31.438130 10031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7884aee96af1427099297ef39e1d65e3" candidate_uuid: "a420200281df441fb040a9357a7cda2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6982669590a45e6a7e4aed0e44b67e6" is_pre_election: true
I20260924 16:14:31.438632 10031 raft_consensus.cc:2454] T 7884aee96af1427099297ef39e1d65e3 P d6982669590a45e6a7e4aed0e44b67e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a420200281df441fb040a9357a7cda2f in term 0.
I20260924 16:14:31.438633 10167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7884aee96af1427099297ef39e1d65e3" candidate_uuid: "a420200281df441fb040a9357a7cda2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5db508082b8e4c2b84d6026dc4e167ba" is_pre_election: true
I20260924 16:14:31.439221 10167 raft_consensus.cc:2454] T 7884aee96af1427099297ef39e1d65e3 P 5db508082b8e4c2b84d6026dc4e167ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a420200281df441fb040a9357a7cda2f in term 0.
I20260924 16:14:31.439299 9819 leader_election.cc:302] T 7884aee96af1427099297ef39e1d65e3 P a420200281df441fb040a9357a7cda2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a420200281df441fb040a9357a7cda2f, d6982669590a45e6a7e4aed0e44b67e6; no voters:
I20260924 16:14:31.439702 10241 raft_consensus.cc:2781] T 7884aee96af1427099297ef39e1d65e3 P a420200281df441fb040a9357a7cda2f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:14:31.439852 10241 raft_consensus.cc:482] T 7884aee96af1427099297ef39e1d65e3 P a420200281df441fb040a9357a7cda2f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:14:31.439961 10241 raft_consensus.cc:3037] T 7884aee96af1427099297ef39e1d65e3 P a420200281df441fb040a9357a7cda2f [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:14:31.442224 10241 raft_consensus.cc:504] T 7884aee96af1427099297ef39e1d65e3 P a420200281df441fb040a9357a7cda2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a420200281df441fb040a9357a7cda2f" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 32953 } } peers { permanent_uuid: "d6982669590a45e6a7e4aed0e44b67e6" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 46021 } } peers { permanent_uuid: "5db508082b8e4c2b84d6026dc4e167ba" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 36979 } }
I20260924 16:14:31.442906 10241 leader_election.cc:288] T 7884aee96af1427099297ef39e1d65e3 P a420200281df441fb040a9357a7cda2f [CANDIDATE]: Term 1 election: Requested vote from peers d6982669590a45e6a7e4aed0e44b67e6 (127.29.234.2:46021), 5db508082b8e4c2b84d6026dc4e167ba (127.29.234.3:36979)
I20260924 16:14:31.443377 10031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7884aee96af1427099297ef39e1d65e3" candidate_uuid: "a420200281df441fb040a9357a7cda2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6982669590a45e6a7e4aed0e44b67e6"
I20260924 16:14:31.443389 10167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7884aee96af1427099297ef39e1d65e3" candidate_uuid: "a420200281df441fb040a9357a7cda2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5db508082b8e4c2b84d6026dc4e167ba"
I20260924 16:14:31.443605 10167 raft_consensus.cc:3037] T 7884aee96af1427099297ef39e1d65e3 P 5db508082b8e4c2b84d6026dc4e167ba [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:14:31.443593 10031 raft_consensus.cc:3037] T 7884aee96af1427099297ef39e1d65e3 P d6982669590a45e6a7e4aed0e44b67e6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:14:31.445982 10167 raft_consensus.cc:2454] T 7884aee96af1427099297ef39e1d65e3 P 5db508082b8e4c2b84d6026dc4e167ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a420200281df441fb040a9357a7cda2f in term 1.
I20260924 16:14:31.446082 10031 raft_consensus.cc:2454] T 7884aee96af1427099297ef39e1d65e3 P d6982669590a45e6a7e4aed0e44b67e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a420200281df441fb040a9357a7cda2f in term 1.
I20260924 16:14:31.446412 9818 leader_election.cc:302] T 7884aee96af1427099297ef39e1d65e3 P a420200281df441fb040a9357a7cda2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5db508082b8e4c2b84d6026dc4e167ba, a420200281df441fb040a9357a7cda2f; no voters:
I20260924 16:14:31.446709 10241 raft_consensus.cc:2781] T 7884aee96af1427099297ef39e1d65e3 P a420200281df441fb040a9357a7cda2f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:14:31.447126 10241 raft_consensus.cc:686] T 7884aee96af1427099297ef39e1d65e3 P a420200281df441fb040a9357a7cda2f [term 1 LEADER]: Becoming Leader. State: Replica: a420200281df441fb040a9357a7cda2f, State: Running, Role: LEADER
I20260924 16:14:31.447664 10241 consensus_queue.cc:237] T 7884aee96af1427099297ef39e1d65e3 P a420200281df441fb040a9357a7cda2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a420200281df441fb040a9357a7cda2f" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 32953 } } peers { permanent_uuid: "d6982669590a45e6a7e4aed0e44b67e6" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 46021 } } peers { permanent_uuid: "5db508082b8e4c2b84d6026dc4e167ba" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 36979 } }
I20260924 16:14:31.452026 9715 catalog_manager.cc:5495] T 7884aee96af1427099297ef39e1d65e3 P a420200281df441fb040a9357a7cda2f reported cstate change: term changed from 0 to 1, leader changed from <none> to a420200281df441fb040a9357a7cda2f (127.29.234.1). New cstate: current_term: 1 leader_uuid: "a420200281df441fb040a9357a7cda2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a420200281df441fb040a9357a7cda2f" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 32953 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6982669590a45e6a7e4aed0e44b67e6" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 46021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5db508082b8e4c2b84d6026dc4e167ba" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 36979 } health_report { overall_health: UNKNOWN } } }
W20260924 16:14:31.478252 10215 tablet.cc:2369] T 7884aee96af1427099297ef39e1d65e3 P 5db508082b8e4c2b84d6026dc4e167ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:14:31.486361 10228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:31.480504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48578 (local address 127.29.234.1:32953)
0924 16:14:31.480714 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 16:14:31.480721 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:14:31.480745 (+ 24us) server_negotiation.cc:407] Connection header received
0924 16:14:31.481025 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:14:31.481029 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:14:31.481116 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:14:31.481274 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:14:31.481677 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:31.482622 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:31.484142 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:31.484551 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:31.484683 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:31.485175 (+ 492us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:14:31.485775 (+ 600us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:14:31.485940 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0924 16:14:31.485997 (+ 57us) server_negotiation.cc:299] Negotiation successful
0924 16:14:31.486076 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 16:14:31.494405 10031 raft_consensus.cc:1261] T 7884aee96af1427099297ef39e1d65e3 P d6982669590a45e6a7e4aed0e44b67e6 [term 1 FOLLOWER]: Refusing update from remote peer a420200281df441fb040a9357a7cda2f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:14:31.494382 10167 raft_consensus.cc:1261] T 7884aee96af1427099297ef39e1d65e3 P 5db508082b8e4c2b84d6026dc4e167ba [term 1 FOLLOWER]: Refusing update from remote peer a420200281df441fb040a9357a7cda2f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:14:31.495520 10241 consensus_queue.cc:1035] T 7884aee96af1427099297ef39e1d65e3 P a420200281df441fb040a9357a7cda2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5db508082b8e4c2b84d6026dc4e167ba" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 36979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:14:31.495844 10244 consensus_queue.cc:1035] T 7884aee96af1427099297ef39e1d65e3 P a420200281df441fb040a9357a7cda2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6982669590a45e6a7e4aed0e44b67e6" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 46021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:14:31.528770 9715 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52946:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:14:31.529057 9715 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52946:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:14:31.532641 9715 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bace56cc062f4d4bb51afd00f6eba991: Sending DeleteTablet for 3 replicas of tablet 7884aee96af1427099297ef39e1d65e3
I20260924 16:14:31.533741 9866 tablet_service.cc:1514] Processing DeleteTablet for tablet 7884aee96af1427099297ef39e1d65e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:14:31 UTC) from {username='kudu', principal='kudu/127.29.234.62@KRBTEST.COM'} at 127.0.0.1:48576
I20260924 16:14:31.533913 10010 tablet_service.cc:1514] Processing DeleteTablet for tablet 7884aee96af1427099297ef39e1d65e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:14:31 UTC) from {username='kudu', principal='kudu/127.29.234.62@KRBTEST.COM'} at 127.0.0.1:55556
I20260924 16:14:31.534199 10147 tablet_service.cc:1514] Processing DeleteTablet for tablet 7884aee96af1427099297ef39e1d65e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:14:31 UTC) from {username='kudu', principal='kudu/127.29.234.62@KRBTEST.COM'} at 127.0.0.1:37464
I20260924 16:14:31.536993 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 9797
I20260924 16:14:31.544697 10259 tablet_replica.cc:330] T 7884aee96af1427099297ef39e1d65e3 P d6982669590a45e6a7e4aed0e44b67e6: stopping tablet replica
I20260924 16:14:31.545284 10259 raft_consensus.cc:2229] T 7884aee96af1427099297ef39e1d65e3 P d6982669590a45e6a7e4aed0e44b67e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:14:31.545935 10259 raft_consensus.cc:2258] T 7884aee96af1427099297ef39e1d65e3 P d6982669590a45e6a7e4aed0e44b67e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:14:31.549052 10259 ts_tablet_manager.cc:1905] T 7884aee96af1427099297ef39e1d65e3 P d6982669590a45e6a7e4aed0e44b67e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:14:31.558032 10258 tablet_replica.cc:330] T 7884aee96af1427099297ef39e1d65e3 P a420200281df441fb040a9357a7cda2f: stopping tablet replica
I20260924 16:14:31.558625 10258 raft_consensus.cc:2229] T 7884aee96af1427099297ef39e1d65e3 P a420200281df441fb040a9357a7cda2f [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:14:31.558724 10260 tablet_replica.cc:330] T 7884aee96af1427099297ef39e1d65e3 P 5db508082b8e4c2b84d6026dc4e167ba: stopping tablet replica
I20260924 16:14:31.559962 10259 ts_tablet_manager.cc:1918] T 7884aee96af1427099297ef39e1d65e3 P d6982669590a45e6a7e4aed0e44b67e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:14:31.560043 10260 raft_consensus.cc:2229] T 7884aee96af1427099297ef39e1d65e3 P 5db508082b8e4c2b84d6026dc4e167ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:14:31.560155 10259 log.cc:1196] T 7884aee96af1427099297ef39e1d65e3 P d6982669590a45e6a7e4aed0e44b67e6: Deleting WAL directory at /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/ts-1/wal/wals/7884aee96af1427099297ef39e1d65e3
I20260924 16:14:31.560637 10260 raft_consensus.cc:2258] T 7884aee96af1427099297ef39e1d65e3 P 5db508082b8e4c2b84d6026dc4e167ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:14:31.560739 10259 ts_tablet_manager.cc:1939] T 7884aee96af1427099297ef39e1d65e3 P d6982669590a45e6a7e4aed0e44b67e6: Deleting consensus metadata
I20260924 16:14:31.562817 10260 ts_tablet_manager.cc:1905] T 7884aee96af1427099297ef39e1d65e3 P 5db508082b8e4c2b84d6026dc4e167ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:14:31.563191 9700 catalog_manager.cc:4841] TS d6982669590a45e6a7e4aed0e44b67e6 (127.29.234.2:46021): tablet 7884aee96af1427099297ef39e1d65e3 (table test-table [id=1f392921510e45c8b6e9c635802dd690]) successfully deleted
Sep 24 16:14:31 dist-test-slave-v5j9 krb5kdc[9229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.234.1@KRBTEST.COM
I20260924 16:14:31.563939 10258 raft_consensus.cc:2258] T 7884aee96af1427099297ef39e1d65e3 P a420200281df441fb040a9357a7cda2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:14:31.566310 10258 ts_tablet_manager.cc:1905] T 7884aee96af1427099297ef39e1d65e3 P a420200281df441fb040a9357a7cda2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:14:31.573483 10228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:31.540891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48584 (local address 127.29.234.1:32953)
0924 16:14:31.541120 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0924 16:14:31.541129 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:14:31.547097 (+ 5968us) server_negotiation.cc:407] Connection header received
0924 16:14:31.547328 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:14:31.547334 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:14:31.547508 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:14:31.547614 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:14:31.554016 (+ 6402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:31.555220 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:31.558576 (+ 3356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:31.559310 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:31.559420 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:31.565263 (+ 5843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:14:31.565277 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:14:31.565296 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:14:31.565379 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:14:31.569627 (+ 4248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:31.570623 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:31.570633 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:31.570638 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:31.570828 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:31.571313 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:31.571318 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:31.571321 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:31.571852 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:14:31.572153 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0924 16:14:31.572774 (+ 621us) server_negotiation.cc:299] Negotiation successful
0924 16:14:31.573066 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 16:14:31.574826 10258 ts_tablet_manager.cc:1918] T 7884aee96af1427099297ef39e1d65e3 P a420200281df441fb040a9357a7cda2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:14:31.575021 10258 log.cc:1196] T 7884aee96af1427099297ef39e1d65e3 P a420200281df441fb040a9357a7cda2f: Deleting WAL directory at /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/ts-0/wal/wals/7884aee96af1427099297ef39e1d65e3
I20260924 16:14:31.575631 10258 ts_tablet_manager.cc:1939] T 7884aee96af1427099297ef39e1d65e3 P a420200281df441fb040a9357a7cda2f: Deleting consensus metadata
I20260924 16:14:31.575735 9927 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:14:31.581044 10260 ts_tablet_manager.cc:1918] T 7884aee96af1427099297ef39e1d65e3 P 5db508082b8e4c2b84d6026dc4e167ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:14:31.581214 10260 log.cc:1196] T 7884aee96af1427099297ef39e1d65e3 P 5db508082b8e4c2b84d6026dc4e167ba: Deleting WAL directory at /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790266323169505-30632-0/minicluster-data/ts-2/wal/wals/7884aee96af1427099297ef39e1d65e3
I20260924 16:14:31.581776 10260 ts_tablet_manager.cc:1939] T 7884aee96af1427099297ef39e1d65e3 P 5db508082b8e4c2b84d6026dc4e167ba: Deleting consensus metadata
I20260924 16:14:31.583374 9699 catalog_manager.cc:4841] TS 5db508082b8e4c2b84d6026dc4e167ba (127.29.234.3:36979): tablet 7884aee96af1427099297ef39e1d65e3 (table test-table [id=1f392921510e45c8b6e9c635802dd690]) successfully deleted
I20260924 16:14:31.678841 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 9797
I20260924 16:14:31.679140 9701 catalog_manager.cc:4841] TS a420200281df441fb040a9357a7cda2f (127.29.234.1:32953): tablet 7884aee96af1427099297ef39e1d65e3 (table test-table [id=1f392921510e45c8b6e9c635802dd690]) successfully deleted
I20260924 16:14:31.697640 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 9939
Sep 24 16:14:31 dist-test-slave-v5j9 krb5kdc[9229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.234.2@KRBTEST.COM
I20260924 16:14:31.713238 10229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:31.698441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55568 (local address 127.29.234.2:46021)
0924 16:14:31.698624 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 16:14:31.698632 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:14:31.698706 (+ 74us) server_negotiation.cc:407] Connection header received
0924 16:14:31.698921 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:14:31.698927 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:14:31.699003 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:14:31.699106 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:14:31.700369 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:31.701112 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:31.702568 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:31.702958 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:31.703094 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:31.707477 (+ 4383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:14:31.707493 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:14:31.707512 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:14:31.707548 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:14:31.710338 (+ 2790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:31.711110 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:31.711118 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:31.711120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:31.711259 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:31.711646 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:31.711651 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:31.711654 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:31.712027 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:14:31.712338 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0924 16:14:31.712657 (+ 319us) server_negotiation.cc:299] Negotiation successful
0924 16:14:31.712884 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 16:14:31.713974 10072 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:14:31.810119 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 9939
I20260924 16:14:31.828675 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 10083
Sep 24 16:14:31 dist-test-slave-v5j9 krb5kdc[9229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.234.3@KRBTEST.COM
I20260924 16:14:31.852782 10232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:14:31.829453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37474 (local address 127.29.234.3:36979)
0924 16:14:31.829623 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 16:14:31.829628 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:14:31.829646 (+ 18us) server_negotiation.cc:407] Connection header received
0924 16:14:31.829856 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:14:31.829861 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:14:31.829928 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:14:31.830014 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:14:31.831617 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:31.832352 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:31.833763 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:14:31.834144 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:14:31.834235 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:14:31.846261 (+ 12026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:14:31.846287 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:14:31.846343 (+ 56us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:14:31.846417 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:14:31.849710 (+ 3293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:31.850524 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:31.850533 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:31.850535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:31.850658 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:14:31.851091 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:14:31.851098 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:14:31.851106 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:14:31.851500 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:14:31.851807 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0924 16:14:31.852161 (+ 354us) server_negotiation.cc:299] Negotiation successful
0924 16:14:31.852440 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 16:14:31.853470 10209 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:14:31.942265 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 10083
I20260924 16:14:31.959584 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 9682
I20260924 16:14:31.960873 9745 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:14:32.074461 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 9682
I20260924 16:14:32.097226 30632 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 16:14:32.433491 30632 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 16:14:32.433733 30632 mini_ranger.cc:66] Stopping Ranger...
I20260924 16:14:32.779173 30632 mini_ranger.cc:68] Stopped Ranger
2026-09-24 16:14:32.779 UTC [9259] LOG: received smart shutdown request
2026-09-24 16:14:32.783 UTC [9259] LOG: background worker "logical replication launcher" (PID 9268) exited with exit code 1
2026-09-24 16:14:32.783 UTC [9263] LOG: shutting down
2026-09-24 16:14:32.954 UTC [9259] LOG: database system is shut down
2026-09-24T16:14:32Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49750 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:14:33 dist-test-slave-v5j9 krb5kdc[10274](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:14:33 dist-test-slave-v5j9 krb5kdc[10274](info): set up 2 sockets
Sep 24 16:14:33 dist-test-slave-v5j9 krb5kdc[10274](info): commencing operation
krb5kdc: starting...
W20260924 16:14:34.225099 30632 mini_kdc.cc:121] Time spent starting KDC: real 1.095s user 0.007s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Sep 24 16:14:34 dist-test-slave-v5j9 krb5kdc[10274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:14:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:14:34Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.29.234.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.29.234.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.29.234.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/krb5kdc/rangeradmin_127.29.234.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.29.234.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/krb5kdc/rangeradmin_127.29.234.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.29.234.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.29.234.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.29.234.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/krb5kdc/rangerlookup_127.29.234.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.29.234.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/krb5kdc/rangerlookup_127.29.234.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.29.234.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.29.234.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.29.234.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/krb5kdc/HTTP_127.29.234.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.29.234.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/krb5kdc/HTTP_127.29.234.20@KRBTEST.COM.keytab.
I20260924 16:14:34.558283 30632 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskQurfSc/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/postgres -l logfile start
2026-09-24 16:14:37.105 UTC [10304] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 16:14:37.106 UTC [10304] LOG: listening on IPv4 address "127.29.234.20", port 55407
2026-09-24 16:14:37.109 UTC [10304] LOG: listening on Unix socket "/tmp/.s.PGSQL.55407"
2026-09-24 16:14:37.131 UTC [10307] LOG: database system was shut down at 2026-09-24 16:14:35 UTC
2026-09-24 16:14:37.134 UTC [10304] LOG: database system is ready to accept connections
I20260924 16:14:38.116943 30632 mini_postgres.cc:96] Postgres bound to 55407
2026-09-24 16:14:38.134 UTC [10315] FATAL: database "slave" does not exist
127.29.234.20:55407 - accepting connections
I20260924 16:14:38.135108 30632 mini_ranger.cc:161] Starting Ranger...
I20260924 16:14:38.161213 30632 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 16:14:38.667197 30632 mini_ranger.cc:87] Created ranger Postgres database
I20260924 16:14:38.667320 30632 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ranger-admin
2026-09-24 16:14:38,976 [I] DB FLAVOR :POSTGRES
2026-09-24 16:14:38,976 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 16:14:38,977 [I] Checking connection..
2026-09-24 16:14:38,977 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQurfSc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQurfSc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.234.20:55407/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 16:14:39,221 [I] Checking connection passed.
2026-09-24 16:14:39,222 [I] --------- Verifying version history table ---------
2026-09-24 16:14:39,222 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQurfSc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQurfSc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.234.20:55407/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 16:14:39,488 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 16:14:39,488 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQurfSc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQurfSc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.234.20:55407/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 16:14:39,735 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 16:14:39,736 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 16:14:39,736 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQurfSc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQurfSc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.234.20:55407/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskQurfSc/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 16:14:39.976 UTC [10381] WARNING: there is no transaction in progress
2026-09-24 16:14:39,983 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 16:14:39,983 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQurfSc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQurfSc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.234.20:55407/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 16:14:40,224 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 16:14:40,224 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 16:14:40,224 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQurfSc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQurfSc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.234.20:55407/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 16:14:40,451 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQurfSc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQurfSc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.234.20:55407/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 16:14:40,688 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQurfSc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQurfSc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.234.20:55407/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-v5j9.c.gcp-upstream.internal','N') ;"
2026-09-24 16:14:40,940 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQurfSc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQurfSc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.234.20:55407/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 16:14:41,216 [I] Table x_portal_user does not exist in database ranger
2026-09-24 16:14:41,217 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQurfSc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQurfSc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.234.20:55407/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 16:14:41,453 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 16:14:41,454 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQurfSc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQurfSc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.234.20:55407/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 16:14:41,713 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQurfSc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQurfSc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.234.20:55407/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 16:14:41,967 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 16:14:41,967 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQurfSc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQurfSc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.234.20:55407/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskQurfSc/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 16:14:42.922 UTC [10490] WARNING: there is no transaction in progress
2026-09-24 16:14:42.939 UTC [10490] WARNING: there is no transaction in progress
2026-09-24 16:14:42.956 UTC [10490] WARNING: there is no transaction in progress
2026-09-24 16:14:42.972 UTC [10490] WARNING: there is no transaction in progress
2026-09-24 16:14:42.988 UTC [10490] WARNING: there is no transaction in progress
2026-09-24 16:14:43.010 UTC [10490] WARNING: there is no transaction in progress
2026-09-24 16:14:43.066 UTC [10490] WARNING: there is no transaction in progress
2026-09-24 16:14:43.073 UTC [10490] WARNING: there is no transaction in progress
2026-09-24 16:14:43.087 UTC [10490] WARNING: there is no transaction in progress
2026-09-24 16:14:43.097 UTC [10490] WARNING: there is no transaction in progress
2026-09-24 16:14:43.107 UTC [10490] WARNING: there is no transaction in progress
2026-09-24 16:14:43.121 UTC [10490] WARNING: there is no transaction in progress
2026-09-24 16:14:43.131 UTC [10490] WARNING: there is no transaction in progress
2026-09-24 16:14:43.137 UTC [10490] WARNING: there is no transaction in progress
2026-09-24 16:14:43.145 UTC [10490] WARNING: there is no transaction in progress
2026-09-24 16:14:43.151 UTC [10490] WARNING: there is no transaction in progress
2026-09-24 16:14:43,786 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 16:14:43,787 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQurfSc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQurfSc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.234.20:55407/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 16:14:44,045 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 16:14:44,046 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQurfSc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQurfSc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.234.20:55407/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 16:14:44,293 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 16:14:44,294 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQurfSc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQurfSc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.234.20:55407/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 16:14:44,528 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 16:14:44,528 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQurfSc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQurfSc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.234.20:55407/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 16:14:44,783 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQurfSc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQurfSc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.234.20:55407/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 16:14:45,044 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQurfSc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQurfSc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.234.20:55407/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-v5j9.c.gcp-upstream.internal';"
2026-09-24 16:14:45,302 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 16:14:45,302 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQurfSc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQurfSc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.234.20:55407/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 16:14:45,550 [I] DB_PATCHES have already been applied
I20260924 16:14:45.555991 30632 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ranger-admin:/tmp/dist-test-taskQurfSc/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskQurfSc/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskQurfSc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQurfSc/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 16:14:45.556083 30632 mini_ranger.cc:193] Using host: 127.29.234.20
I20260924 16:14:45.571821 30632 mini_ranger.cc:227] Ranger admin URL: http://127.29.234.20:33439
Sep 24, 2026 4:14:45 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 4:14:45 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 4:14:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 4:14:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 4:14:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 4:14:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 4:14:46 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 16:14:46.740693 30632 mini_ranger.cc:160] Time spent starting Ranger: real 8.606s user 0.001s sys 0.040s
Sep 24, 2026 4:15:07 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 4:15:07 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Sep 24, 2026 4:15:07 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 4:15:07 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 4:15:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 4:15:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 4:15:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 4:15:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 4:15:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 4:15:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 4:15:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 4:15:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 4:15:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 4:15:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 4:15:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 4:15:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 4:15:07 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 4:15:08 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 16:15:08.964350 30632 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.29.234.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.29.234.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.29.234.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/krb5kdc/rangerkms_127.29.234.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.29.234.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/krb5kdc/rangerkms_127.29.234.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.29.234.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.29.234.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.29.234.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/krb5kdc/HTTP_127.29.234.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.29.234.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/krb5kdc/HTTP_127.29.234.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 16:15:09 dist-test-slave-v5j9 krb5kdc[10274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266509, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 16:15:09.136816 30632 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 16:15:09.157435 30632 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 16:15:10.057581 30632 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 16:15:10.057682 30632 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ranger-kms
I20260924 16:15:10.057716 30632 mini_ranger_kms.cc:227] Using postgres at 127.29.234.20:55407
2026-09-24 16:15:10,414 [I] DB FLAVOR :POSTGRES
2026-09-24 16:15:10,415 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 16:15:10,415 [I] Checking connection
2026-09-24 16:15:10,415 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQurfSc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.234.20:55407/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 16:15:10,643 [I] connection success
2026-09-24 16:15:10,643 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 16:15:10,643 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 16:15:10,643 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQurfSc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.234.20:55407/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 16:15:10,885 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 16:15:10,886 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 16:15:10,886 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 16:15:10,886 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQurfSc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.234.20:55407/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 16:15:11,144 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 16:15:11.620020 30632 mini_ranger_kms.cc:310] Created kms service
I20260924 16:15:11.796979 30632 mini_ranger_kms.cc:326] Created kudu user
I20260924 16:15:11.844393 30632 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 16:15:11 dist-test-slave-v5j9 krb5kdc[10274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266509, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.29.234.20@KRBTEST.COM
I20260924 16:15:12.250680 30632 mini_ranger_kms.cc:384] Added ranger policy
I20260924 16:15:12.251037 30632 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ranger-kms:/tmp/dist-test-taskQurfSc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQurfSc/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskQurfSc/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 16:15:12.251120 30632 mini_ranger_kms.cc:240] Using host: 127.29.234.20
I20260924 16:15:12.266907 30632 mini_ranger_kms.cc:276] Ranger KMS PID: 10674
I20260924 16:15:12.267035 30632 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.29.234.20:50657
Sep 24, 2026 4:15:13 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 4:15:13 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 4:15:13 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 4:15:13 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 4:15:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 4:15:14 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 16:15:14.591388 30632 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.455s user 0.100s sys 0.268s
I20260924 16:15:14.591657 30632 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 16:15:14.591842 30632 mini_ranger_kms.cc:401] 127.29.234.20:50657/kms/v1/keys
Sep 24 16:15:14 dist-test-slave-v5j9 krb5kdc[10274](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790266514, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.29.234.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 16:15:15 dist-test-slave-v5j9 krb5kdc[10274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266514, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.29.234.20@KRBTEST.COM for HTTP/127.29.234.20@KRBTEST.COM
Sep 24 16:15:15 dist-test-slave-v5j9 krb5kdc[10274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266514, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.29.234.20@KRBTEST.COM for HTTP/127.29.234.20@KRBTEST.COM
Sep 24 16:15:17 dist-test-slave-v5j9 krb5kdc[10274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.29.234.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab.
Entry for principal kudu/127.29.234.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab.
I20260924 16:15:18.015298 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:34419
--webserver_interface=127.29.234.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab
--principal=kudu/127.29.234.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:33053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.234.62:34419
--ranger_config_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.234.20:50657/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:15:18.352810 10726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:15:18.353173 10726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:15:18.353410 10726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:15:18.360980 10726 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:15:18.361130 10726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:15:18.361235 10726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:15:18.361339 10726 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:15:18.361434 10726 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:15:18.371678 10726 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:33053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/master-0/wal
--ranger_kms_url=127.29.234.20:50657/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.29.234.62:34419
--ranger_config_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab
--principal=kudu/127.29.234.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:34419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.234.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:15:18.373133 10726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:15:18.374619 10726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:15:18.387112 10734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:18.387249 10732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:18.387272 10731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:15:18.390689 10726 server_base.cc:1034] running on GCE node
I20260924 16:15:18.391530 10726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:15:18.392984 10726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:15:18.394222 10726 hybrid_clock.cc:648] HybridClock initialized: now 1790266518394163 us; error 53 us; skew 500 ppm
I20260924 16:15:18.394615 10726 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:15:18 dist-test-slave-v5j9 krb5kdc[10274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266518, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:15:18.403880 10726 init.cc:377] Logged in from keytab as kudu/127.29.234.62@KRBTEST.COM (short username kudu)
I20260924 16:15:18.405638 10726 webserver.cc:460] Webserver started at http://127.29.234.62:40281/ using document root <none> and password file <none>
I20260924 16:15:18.406387 10726 fs_manager.cc:359] Metadata directory not provided
I20260924 16:15:18.406533 10726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:15:18.406867 10726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 16:15:18 dist-test-slave-v5j9 krb5kdc[10274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266518, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.62@KRBTEST.COM for HTTP/127.29.234.20@KRBTEST.COM
I20260924 16:15:18.464071 10726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/master-0/data/instance:
uuid: "10330b3a301d4583b27b45d298eddf4e"
format_stamp: "Formatted at 2026-09-24 16:15:18 on dist-test-slave-v5j9"
server_key: "73c7ac072fc81c2ffcda0fe9910c9392"
server_key_iv: "83700b1193cbb7d49192113f29c4056a"
server_key_version: "kuduclusterkey@0"
I20260924 16:15:18.464960 10726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/master-0/wal/instance:
uuid: "10330b3a301d4583b27b45d298eddf4e"
format_stamp: "Formatted at 2026-09-24 16:15:18 on dist-test-slave-v5j9"
server_key: "73c7ac072fc81c2ffcda0fe9910c9392"
server_key_iv: "83700b1193cbb7d49192113f29c4056a"
server_key_version: "kuduclusterkey@0"
I20260924 16:15:18.470383 10726 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 16:15:18.489971 10741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:18.491519 10726 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260924 16:15:18.491721 10726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/master-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/master-0/wal
uuid: "10330b3a301d4583b27b45d298eddf4e"
format_stamp: "Formatted at 2026-09-24 16:15:18 on dist-test-slave-v5j9"
server_key: "73c7ac072fc81c2ffcda0fe9910c9392"
server_key_iv: "83700b1193cbb7d49192113f29c4056a"
server_key_version: "kuduclusterkey@0"
I20260924 16:15:18.491971 10726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:15:18.512081 10726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:15:18.515776 10726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:15:18.516106 10726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:15:18.534797 10726 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.62:34419
I20260924 16:15:18.534808 10793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.62:34419 every 8 connection(s)
I20260924 16:15:18.536248 10726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
I20260924 16:15:18.539997 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 10726
I20260924 16:15:18.540237 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/master-0/wal/instance
I20260924 16:15:18.540248 10794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:15:18.540943 30632 external_mini_cluster.cc:1428] Setting key 59ed862d05e23605d6f025c3bb26b9b8
I20260924 16:15:18.554684 10794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10330b3a301d4583b27b45d298eddf4e: Bootstrap starting.
Sep 24 16:15:18 dist-test-slave-v5j9 krb5kdc[10274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:15:18.557963 10794 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10330b3a301d4583b27b45d298eddf4e: Neither blocks nor log segments found. Creating new log.
I20260924 16:15:18.559355 10794 log.cc:824] T 00000000000000000000000000000000 P 10330b3a301d4583b27b45d298eddf4e: Log is configured to *not* fsync() on all Append() calls
I20260924 16:15:18.562635 10794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10330b3a301d4583b27b45d298eddf4e: No bootstrap required, opened a new log
I20260924 16:15:18.565873 10797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:18.543154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42670 (local address 127.29.234.62:34419)
0924 16:15:18.544017 (+ 863us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:18.544032 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:18.544085 (+ 53us) server_negotiation.cc:407] Connection header received
0924 16:15:18.546982 (+ 2897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:18.547013 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:18.547551 (+ 538us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:18.548026 (+ 475us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:18.549250 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:18.551331 (+ 2081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:18.552652 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:18.553702 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:18.553896 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:18.557680 (+ 3784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:18.557694 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:18.557715 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:15:18.557785 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:18.561526 (+ 3741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:18.562176 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:18.562184 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:18.562192 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:18.562395 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:18.562735 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:18.562740 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:18.562743 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:18.563448 (+ 705us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:18.563733 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:18.564072 (+ 339us) server_negotiation.cc:299] Negotiation successful
0924 16:15:18.564474 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":350,"threads_started":1}
I20260924 16:15:18.572021 10794 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10330b3a301d4583b27b45d298eddf4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10330b3a301d4583b27b45d298eddf4e" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 34419 } }
I20260924 16:15:18.572377 10794 raft_consensus.cc:374] T 00000000000000000000000000000000 P 10330b3a301d4583b27b45d298eddf4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:15:18.572456 10794 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10330b3a301d4583b27b45d298eddf4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10330b3a301d4583b27b45d298eddf4e, State: Initialized, Role: FOLLOWER
I20260924 16:15:18.573024 10794 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10330b3a301d4583b27b45d298eddf4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10330b3a301d4583b27b45d298eddf4e" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 34419 } }
I20260924 16:15:18.573223 10794 raft_consensus.cc:388] T 00000000000000000000000000000000 P 10330b3a301d4583b27b45d298eddf4e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:15:18.573319 10794 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10330b3a301d4583b27b45d298eddf4e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:15:18.573509 10794 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10330b3a301d4583b27b45d298eddf4e [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:15:18.575855 10794 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10330b3a301d4583b27b45d298eddf4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10330b3a301d4583b27b45d298eddf4e" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 34419 } }
I20260924 16:15:18.576428 10794 leader_election.cc:302] T 00000000000000000000000000000000 P 10330b3a301d4583b27b45d298eddf4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 10330b3a301d4583b27b45d298eddf4e; no voters:
I20260924 16:15:18.576944 10794 leader_election.cc:288] T 00000000000000000000000000000000 P 10330b3a301d4583b27b45d298eddf4e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:15:18.577099 10799 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10330b3a301d4583b27b45d298eddf4e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:15:18.577790 10799 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10330b3a301d4583b27b45d298eddf4e [term 1 LEADER]: Becoming Leader. State: Replica: 10330b3a301d4583b27b45d298eddf4e, State: Running, Role: LEADER
I20260924 16:15:18.578359 10799 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10330b3a301d4583b27b45d298eddf4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10330b3a301d4583b27b45d298eddf4e" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 34419 } }
I20260924 16:15:18.579139 10794 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10330b3a301d4583b27b45d298eddf4e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:15:18.580996 10801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10330b3a301d4583b27b45d298eddf4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10330b3a301d4583b27b45d298eddf4e. Latest consensus state: current_term: 1 leader_uuid: "10330b3a301d4583b27b45d298eddf4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10330b3a301d4583b27b45d298eddf4e" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 34419 } } }
I20260924 16:15:18.581391 10801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10330b3a301d4583b27b45d298eddf4e [sys.catalog]: This master's current role is: LEADER
I20260924 16:15:18.581632 10800 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10330b3a301d4583b27b45d298eddf4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "10330b3a301d4583b27b45d298eddf4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10330b3a301d4583b27b45d298eddf4e" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 34419 } } }
I20260924 16:15:18.582028 10800 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10330b3a301d4583b27b45d298eddf4e [sys.catalog]: This master's current role is: LEADER
I20260924 16:15:18.587391 10794 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 16:15:18.589224 10804 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:15:18.601114 10804 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:15:18.611001 10804 catalog_manager.cc:1347] Generated new cluster ID: 0e72f217e0ea4dff97106ed2004bc219
I20260924 16:15:18.611131 10804 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:15:18.651127 10804 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:15:18.652595 10804 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:15:18.660967 10804 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 10330b3a301d4583b27b45d298eddf4e: Generated new TSK 0
I20260924 16:15:18.662135 10804 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.234.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab.
Entry for principal kudu/127.29.234.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab.
I20260924 16:15:19.439244 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.234.20:50657/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.1:0
--local_ip_for_outbound_sockets=127.29.234.1
--webserver_interface=127.29.234.1
--webserver_port=0
--tserver_master_addrs=127.29.234.62:34419
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab
--principal=kudu/127.29.234.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:33053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 16:15:19 dist-test-slave-v5j9 krb5kdc[10274](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790266519, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 16:15:19.863145 10839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:15:19.863494 10839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:15:19.863670 10839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:15:19.871505 10839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:15:19.871678 10839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:15:19.871858 10839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.1
I20260924 16:15:19.882457 10839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:33053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.29.234.20:50657/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab
--principal=kudu/127.29.234.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.234.1
--webserver_port=0
--tserver_master_addrs=127.29.234.62:34419
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.1
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:15:19.883948 10839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:15:19.885509 10839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:15:19.903522 10850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:19.903499 10851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:15:19.912915 10839 server_base.cc:1034] running on GCE node
W20260924 16:15:19.913369 10853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:15:19.914364 10839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:15:19.915781 10839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:15:19.917044 10839 hybrid_clock.cc:648] HybridClock initialized: now 1790266519916972 us; error 68 us; skew 500 ppm
I20260924 16:15:19.917474 10839 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:15:19 dist-test-slave-v5j9 krb5kdc[10274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266519, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:15:19.920753 10839 init.cc:377] Logged in from keytab as kudu/127.29.234.1@KRBTEST.COM (short username kudu)
I20260924 16:15:19.930804 10839 webserver.cc:460] Webserver started at http://127.29.234.1:39285/ using document root <none> and password file <none>
I20260924 16:15:19.931618 10839 fs_manager.cc:359] Metadata directory not provided
I20260924 16:15:19.931779 10839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:15:19.932112 10839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 16:15:19 dist-test-slave-v5j9 krb5kdc[10274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266519, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.1@KRBTEST.COM for HTTP/127.29.234.20@KRBTEST.COM
I20260924 16:15:19.965922 10839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-0/data/instance:
uuid: "c5d39193651b4ebeb5b0ee4a56f32173"
format_stamp: "Formatted at 2026-09-24 16:15:19 on dist-test-slave-v5j9"
server_key: "c778fc799ce0da748a0b759290e0e7d4"
server_key_iv: "55abf7b230bdf2420a8e12b34d3873a5"
server_key_version: "kuduclusterkey@0"
I20260924 16:15:19.967303 10839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-0/wal/instance:
uuid: "c5d39193651b4ebeb5b0ee4a56f32173"
format_stamp: "Formatted at 2026-09-24 16:15:19 on dist-test-slave-v5j9"
server_key: "c778fc799ce0da748a0b759290e0e7d4"
server_key_iv: "55abf7b230bdf2420a8e12b34d3873a5"
server_key_version: "kuduclusterkey@0"
I20260924 16:15:19.979629 10839 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.003s sys 0.008s
I20260924 16:15:20.018187 10860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:20.027053 10839 fs_manager.cc:725] Time spent opening block manager: real 0.026s user 0.008s sys 0.008s
I20260924 16:15:20.027417 10839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-0/wal
uuid: "c5d39193651b4ebeb5b0ee4a56f32173"
format_stamp: "Formatted at 2026-09-24 16:15:19 on dist-test-slave-v5j9"
server_key: "c778fc799ce0da748a0b759290e0e7d4"
server_key_iv: "55abf7b230bdf2420a8e12b34d3873a5"
server_key_version: "kuduclusterkey@0"
I20260924 16:15:20.027806 10839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:15:20.060562 10839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:15:20.065131 10839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:15:20.065534 10839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:15:20.077862 10839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:15:20.082504 10839 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:15:20.082693 10839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:20.082904 10839 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:15:20.083026 10839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:20.180409 10839 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.1:35129
I20260924 16:15:20.180478 10975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.1:35129 every 8 connection(s)
I20260924 16:15:20.182282 10839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
I20260924 16:15:20.183023 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 10839
I20260924 16:15:20.183308 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-0/wal/instance
I20260924 16:15:20.184415 30632 external_mini_cluster.cc:1428] Setting key ed52d653b6caf05ea0215fb8bacacdfe
Sep 24 16:15:20 dist-test-slave-v5j9 krb5kdc[10274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266519, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.1@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.234.2@KRBTEST.COM; defaulting to no policy
I20260924 16:15:20.222115 10979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:20.187783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.1:34739 (local address 127.29.234.62:34419)
0924 16:15:20.201864 (+ 14081us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:20.201871 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:20.201893 (+ 22us) server_negotiation.cc:407] Connection header received
0924 16:15:20.201959 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:20.201964 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:20.202065 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:20.202184 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:20.204065 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:20.205327 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:20.208087 (+ 2760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:20.208893 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:20.209029 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:20.213972 (+ 4943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:20.213981 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:20.213989 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:15:20.214048 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:20.218144 (+ 4096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:20.219220 (+ 1076us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:20.219226 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:20.219230 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:20.219392 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:20.220018 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:20.220025 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:20.220030 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:20.220461 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:20.220608 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:20.221502 (+ 894us) server_negotiation.cc:299] Negotiation successful
0924 16:15:20.221730 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13947,"thread_start_us":13834,"threads_started":1}
I20260924 16:15:20.223500 10978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:20.186100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:34419 (local address 127.29.234.1:34739)
0924 16:15:20.193969 (+ 7869us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:20.194023 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:20.197253 (+ 3230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:20.202496 (+ 5243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:20.202511 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:20.203247 (+ 736us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:15:20.203876 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:20.203894 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:20.205555 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:20.205564 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:20.207889 (+ 2325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:20.207916 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:20.209403 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:20.209414 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:20.209758 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:20.210568 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:15:20.210605 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:15:20.213670 (+ 3065us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:15:20.218344 (+ 4674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:15:20.218355 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:15:20.218379 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:15:20.219046 (+ 667us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:15:20.219542 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:15:20.219550 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:15:20.219555 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:15:20.219870 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:15:20.220700 (+ 830us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:15:20.220711 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:15:20.221102 (+ 391us) client_negotiation.cc:769] Sending connection context
0924 16:15:20.221424 (+ 322us) client_negotiation.cc:240] Negotiation successful
0924 16:15:20.221983 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7506,"thread_start_us":7262,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.2@KRBTEST.COM" created.
I20260924 16:15:20.225455 10976 heartbeater.cc:347] Connected to a master server at 127.29.234.62:34419
I20260924 16:15:20.225777 10976 heartbeater.cc:464] Registering TS with master...
I20260924 16:15:20.226692 10976 heartbeater.cc:511] Master 127.29.234.62:34419 requested a full tablet report, sending...
I20260924 16:15:20.230662 10758 ts_manager.cc:194] Registered new tserver with Master: c5d39193651b4ebeb5b0ee4a56f32173 (127.29.234.1:35129)
I20260924 16:15:20.233075 10758 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.234.1@KRBTEST.COM'} at 127.29.234.1:34739
Sep 24 16:15:20 dist-test-slave-v5j9 krb5kdc[10274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266519, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.62@KRBTEST.COM for HTTP/127.29.234.20@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab.
Entry for principal kudu/127.29.234.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab.
I20260924 16:15:20.278635 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.234.20:50657/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.2:0
--local_ip_for_outbound_sockets=127.29.234.2
--webserver_interface=127.29.234.2
--webserver_port=0
--tserver_master_addrs=127.29.234.62:34419
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab
--principal=kudu/127.29.234.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:33053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 16:15:20 dist-test-slave-v5j9 krb5kdc[10274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266519, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.62@KRBTEST.COM for HTTP/127.29.234.20@KRBTEST.COM
W20260924 16:15:20.764906 10982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:15:20.765437 10982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:15:20.765643 10982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:15:20.777338 10982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:15:20.777619 10982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:15:20.777861 10982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.2
I20260924 16:15:20.794005 10982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:33053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.29.234.20:50657/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab
--principal=kudu/127.29.234.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.234.2
--webserver_port=0
--tserver_master_addrs=127.29.234.62:34419
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.2
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:15:20.795948 10982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:15:20.797765 10982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:15:20.835649 10990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:20.837939 10992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:20.838500 10989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:15:20.843474 10982 server_base.cc:1034] running on GCE node
I20260924 16:15:20.844221 10982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:15:20.845372 10982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:15:20.846635 10982 hybrid_clock.cc:648] HybridClock initialized: now 1790266520846534 us; error 120 us; skew 500 ppm
I20260924 16:15:20.846992 10982 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:15:20 dist-test-slave-v5j9 krb5kdc[10274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266520, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:15:20.850167 10982 init.cc:377] Logged in from keytab as kudu/127.29.234.2@KRBTEST.COM (short username kudu)
I20260924 16:15:20.872103 10982 webserver.cc:460] Webserver started at http://127.29.234.2:42339/ using document root <none> and password file <none>
I20260924 16:15:20.873080 10982 fs_manager.cc:359] Metadata directory not provided
I20260924 16:15:20.873240 10982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:15:20.873636 10982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 16:15:20 dist-test-slave-v5j9 krb5kdc[10274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266520, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.2@KRBTEST.COM for HTTP/127.29.234.20@KRBTEST.COM
I20260924 16:15:20.889938 10982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-1/data/instance:
uuid: "0190b44255744fdd99cfb02c0951ec26"
format_stamp: "Formatted at 2026-09-24 16:15:20 on dist-test-slave-v5j9"
server_key: "b8698bb8b7ae509e9730dd8619ece987"
server_key_iv: "2e7665e30623e2a074ea7bf867d4701e"
server_key_version: "kuduclusterkey@0"
I20260924 16:15:20.891003 10982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-1/wal/instance:
uuid: "0190b44255744fdd99cfb02c0951ec26"
format_stamp: "Formatted at 2026-09-24 16:15:20 on dist-test-slave-v5j9"
server_key: "b8698bb8b7ae509e9730dd8619ece987"
server_key_iv: "2e7665e30623e2a074ea7bf867d4701e"
server_key_version: "kuduclusterkey@0"
I20260924 16:15:20.897059 10982 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260924 16:15:20.922075 11006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:20.924176 10982 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.006s sys 0.005s
I20260924 16:15:20.924461 10982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-1/wal
uuid: "0190b44255744fdd99cfb02c0951ec26"
format_stamp: "Formatted at 2026-09-24 16:15:20 on dist-test-slave-v5j9"
server_key: "b8698bb8b7ae509e9730dd8619ece987"
server_key_iv: "2e7665e30623e2a074ea7bf867d4701e"
server_key_version: "kuduclusterkey@0"
I20260924 16:15:20.924835 10982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:15:20.946558 10982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:15:20.952234 10982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:15:20.952654 10982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:15:20.954006 10982 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:15:20.958001 10982 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:15:20.958182 10982 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260924 16:15:20.958366 10982 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:15:20.958459 10982 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:21.014487 10982 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.2:36011
I20260924 16:15:21.014549 11120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.2:36011 every 8 connection(s)
I20260924 16:15:21.016259 10982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
I20260924 16:15:21.026193 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 10982
I20260924 16:15:21.026412 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-1/wal/instance
I20260924 16:15:21.027063 30632 external_mini_cluster.cc:1428] Setting key 9243a1929d847ab4bd1af7ac33c6c3ad
Sep 24 16:15:21 dist-test-slave-v5j9 krb5kdc[10274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266520, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.2@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.234.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.3@KRBTEST.COM" created.
I20260924 16:15:21.088667 11123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:21.020105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:34419 (local address 127.29.234.2:49471)
0924 16:15:21.058940 (+ 38835us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:21.059017 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:21.066981 (+ 7964us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:21.068529 (+ 1548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:21.068542 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:21.069212 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:15:21.070102 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:21.070130 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:21.071720 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:21.071726 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:21.073747 (+ 2021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:21.073763 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:21.075214 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:21.075221 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:21.075629 (+ 408us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:21.076443 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:15:21.076475 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:15:21.079138 (+ 2663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:15:21.083202 (+ 4064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:15:21.083212 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:15:21.083232 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:15:21.083843 (+ 611us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:15:21.084329 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:15:21.084334 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:15:21.084337 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:15:21.084654 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:15:21.085452 (+ 798us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:15:21.085460 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:15:21.085978 (+ 518us) client_negotiation.cc:769] Sending connection context
0924 16:15:21.086314 (+ 336us) client_negotiation.cc:240] Negotiation successful
0924 16:15:21.086896 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38435,"thread_start_us":45136,"threads_started":1}
I20260924 16:15:21.094339 11124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:21.020741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.2:49471 (local address 127.29.234.62:34419)
0924 16:15:21.058108 (+ 37367us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:21.058116 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:21.061405 (+ 3289us) server_negotiation.cc:407] Connection header received
0924 16:15:21.067687 (+ 6282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:21.067694 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:21.067810 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:21.067931 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:21.070334 (+ 2403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:21.071501 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:21.073915 (+ 2414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:21.074538 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:21.074836 (+ 298us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:21.079422 (+ 4586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:21.079429 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:21.079433 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:15:21.079479 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:21.083024 (+ 3545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:21.083988 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:21.083994 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:21.083997 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:21.084189 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:21.084809 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:21.084813 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:21.084816 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:21.085201 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:21.085370 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:21.089235 (+ 3865us) server_negotiation.cc:299] Negotiation successful
0924 16:15:21.089520 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37192,"thread_start_us":37406,"threads_started":1}
I20260924 16:15:21.096947 11121 heartbeater.cc:347] Connected to a master server at 127.29.234.62:34419
I20260924 16:15:21.097358 11121 heartbeater.cc:464] Registering TS with master...
I20260924 16:15:21.098336 11121 heartbeater.cc:511] Master 127.29.234.62:34419 requested a full tablet report, sending...
I20260924 16:15:21.100200 10758 ts_manager.cc:194] Registered new tserver with Master: 0190b44255744fdd99cfb02c0951ec26 (127.29.234.2:36011)
I20260924 16:15:21.101295 10758 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.234.2@KRBTEST.COM'} at 127.29.234.2:49471
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab.
Entry for principal kudu/127.29.234.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab.
I20260924 16:15:21.121992 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.234.20:50657/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.3:0
--local_ip_for_outbound_sockets=127.29.234.3
--webserver_interface=127.29.234.3
--webserver_port=0
--tserver_master_addrs=127.29.234.62:34419
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab
--principal=kudu/127.29.234.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:33053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:15:21.237229 10976 heartbeater.cc:503] Master 127.29.234.62:34419 was elected leader, sending a full tablet report...
W20260924 16:15:21.519881 11127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:15:21.520226 11127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:15:21.520397 11127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:15:21.528126 11127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:15:21.528288 11127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:15:21.528474 11127 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.3
I20260924 16:15:21.539605 11127 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:33053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.29.234.20:50657/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab
--principal=kudu/127.29.234.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.234.3
--webserver_port=0
--tserver_master_addrs=127.29.234.62:34419
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.3
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:15:21.541070 11127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:15:21.542511 11127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:15:21.556133 11132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:21.556165 11133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:21.556078 11135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:15:21.556845 11127 server_base.cc:1034] running on GCE node
I20260924 16:15:21.557530 11127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:15:21.558491 11127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:15:21.559691 11127 hybrid_clock.cc:648] HybridClock initialized: now 1790266521559629 us; error 61 us; skew 500 ppm
I20260924 16:15:21.560030 11127 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:15:21 dist-test-slave-v5j9 krb5kdc[10274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266521, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:15:21.563253 11127 init.cc:377] Logged in from keytab as kudu/127.29.234.3@KRBTEST.COM (short username kudu)
I20260924 16:15:21.564849 11127 webserver.cc:460] Webserver started at http://127.29.234.3:37957/ using document root <none> and password file <none>
I20260924 16:15:21.565615 11127 fs_manager.cc:359] Metadata directory not provided
I20260924 16:15:21.565759 11127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:15:21.566077 11127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 16:15:21 dist-test-slave-v5j9 krb5kdc[10274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266521, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.3@KRBTEST.COM for HTTP/127.29.234.20@KRBTEST.COM
I20260924 16:15:21.580271 11127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-2/data/instance:
uuid: "6712eda39e82456db9a5d7e53c096263"
format_stamp: "Formatted at 2026-09-24 16:15:21 on dist-test-slave-v5j9"
server_key: "c4a619c430fb7a5115a626e31fea1622"
server_key_iv: "55fdb6fed2ebd721c917562af38fc6a0"
server_key_version: "kuduclusterkey@0"
I20260924 16:15:21.581121 11127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-2/wal/instance:
uuid: "6712eda39e82456db9a5d7e53c096263"
format_stamp: "Formatted at 2026-09-24 16:15:21 on dist-test-slave-v5j9"
server_key: "c4a619c430fb7a5115a626e31fea1622"
server_key_iv: "55fdb6fed2ebd721c917562af38fc6a0"
server_key_version: "kuduclusterkey@0"
I20260924 16:15:21.586737 11127 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 16:15:21.602821 11142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:21.604343 11127 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:15:21.604537 11127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-2/wal
uuid: "6712eda39e82456db9a5d7e53c096263"
format_stamp: "Formatted at 2026-09-24 16:15:21 on dist-test-slave-v5j9"
server_key: "c4a619c430fb7a5115a626e31fea1622"
server_key_iv: "55fdb6fed2ebd721c917562af38fc6a0"
server_key_version: "kuduclusterkey@0"
I20260924 16:15:21.604772 11127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:15:21.624360 11127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:15:21.628031 11127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:15:21.628319 11127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:15:21.629281 11127 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:15:21.630640 11127 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:15:21.630766 11127 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:21.630919 11127 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:15:21.630991 11127 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:21.664115 11127 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.3:35745
I20260924 16:15:21.664160 11257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.3:35745 every 8 connection(s)
I20260924 16:15:21.665498 11127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
I20260924 16:15:21.666960 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 11127
I20260924 16:15:21.667140 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-2/wal/instance
I20260924 16:15:21.667640 30632 external_mini_cluster.cc:1428] Setting key ee8c33ee1ad1507b3f8c0cc935c03c08
Sep 24 16:15:21 dist-test-slave-v5j9 krb5kdc[10274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266521, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.3@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:15:21.686390 11260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:21.668145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.3:55349 (local address 127.29.234.62:34419)
0924 16:15:21.668673 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:21.668681 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:21.669867 (+ 1186us) server_negotiation.cc:407] Connection header received
0924 16:15:21.672108 (+ 2241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:21.672113 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:21.672204 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:21.672330 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:21.673886 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:21.674608 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:21.676161 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:21.676602 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:21.676747 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:21.680233 (+ 3486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:21.680238 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:21.680244 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:15:21.680293 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:21.683355 (+ 3062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:21.684222 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:21.684226 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:21.684228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:21.684346 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:21.684794 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:21.684799 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:21.684802 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:21.685061 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:21.685171 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:21.685955 (+ 784us) server_negotiation.cc:299] Negotiation successful
0924 16:15:21.686141 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":232,"threads_started":1}
I20260924 16:15:21.687366 11261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:21.668774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:34419 (local address 127.29.234.3:55349)
0924 16:15:21.669691 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:21.669748 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:21.671830 (+ 2082us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:21.672586 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:21.672595 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:21.673035 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:15:21.673716 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:21.673733 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:21.674766 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:21.674772 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:21.676008 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:21.676029 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:21.676829 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:21.676836 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:21.677085 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:21.677840 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:15:21.677861 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:15:21.679975 (+ 2114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:15:21.683498 (+ 3523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:15:21.683507 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:15:21.683536 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:15:21.684085 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:15:21.684436 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:15:21.684439 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:15:21.684441 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:15:21.684683 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:15:21.685171 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:15:21.685176 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:15:21.685549 (+ 373us) client_negotiation.cc:769] Sending connection context
0924 16:15:21.685809 (+ 260us) client_negotiation.cc:240] Negotiation successful
0924 16:15:21.686194 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":354,"threads_started":1}
I20260924 16:15:21.688680 11258 heartbeater.cc:347] Connected to a master server at 127.29.234.62:34419
I20260924 16:15:21.688999 11258 heartbeater.cc:464] Registering TS with master...
I20260924 16:15:21.689746 11258 heartbeater.cc:511] Master 127.29.234.62:34419 requested a full tablet report, sending...
I20260924 16:15:21.691231 10758 ts_manager.cc:194] Registered new tserver with Master: 6712eda39e82456db9a5d7e53c096263 (127.29.234.3:35745)
I20260924 16:15:21.692374 10758 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.234.3@KRBTEST.COM'} at 127.29.234.3:55349
I20260924 16:15:21.695981 30632 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:15:21.697748 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 10839
Sep 24 16:15:21 dist-test-slave-v5j9 krb5kdc[10274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.234.1@KRBTEST.COM
I20260924 16:15:21.714084 11263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:21.698728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53956 (local address 127.29.234.1:35129)
0924 16:15:21.699404 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:21.699410 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:21.699434 (+ 24us) server_negotiation.cc:407] Connection header received
0924 16:15:21.699489 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:21.699495 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:21.699673 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:21.699794 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:21.701539 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:21.702603 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:21.704045 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:21.704544 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:21.704642 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:21.708698 (+ 4056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:21.708708 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:21.708722 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:15:21.708752 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:21.711663 (+ 2911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:21.712251 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:21.712257 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:21.712259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:21.712397 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:21.712690 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:21.712693 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:21.712695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:21.713010 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:21.713248 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:21.713501 (+ 253us) server_negotiation.cc:299] Negotiation successful
0924 16:15:21.713696 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":336,"threads_started":1}
I20260924 16:15:21.715371 10971 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:15:21.797240 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 10839
I20260924 16:15:21.814016 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 10982
Sep 24 16:15:21 dist-test-slave-v5j9 krb5kdc[10274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.234.2@KRBTEST.COM
I20260924 16:15:21.830123 11265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:21.814925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54884 (local address 127.29.234.2:36011)
0924 16:15:21.815579 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:21.815587 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:21.815614 (+ 27us) server_negotiation.cc:407] Connection header received
0924 16:15:21.815679 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:21.815686 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:21.815927 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:21.816064 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:21.817417 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:21.818463 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:21.819833 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:21.820394 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:21.820490 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:21.824695 (+ 4205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:21.824707 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:21.824723 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:15:21.824755 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:21.827464 (+ 2709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:21.828116 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:21.828125 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:21.828133 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:21.828294 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:21.828614 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:21.828617 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:21.828619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:21.828970 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:21.829244 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:21.829554 (+ 310us) server_negotiation.cc:299] Negotiation successful
0924 16:15:21.829817 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":374,"threads_started":1}
I20260924 16:15:21.831637 11116 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:15:21.916374 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 10982
I20260924 16:15:21.932708 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 11127
Sep 24 16:15:21 dist-test-slave-v5j9 krb5kdc[10274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.234.3@KRBTEST.COM
I20260924 16:15:21.949272 11267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:21.933832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53690 (local address 127.29.234.3:35745)
0924 16:15:21.934447 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:21.934454 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:21.934483 (+ 29us) server_negotiation.cc:407] Connection header received
0924 16:15:21.934544 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:21.934553 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:21.934772 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:21.934946 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:21.936354 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:21.937450 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:21.938736 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:21.939305 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:21.939465 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:21.943794 (+ 4329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:21.943805 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:21.943821 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:15:21.943854 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:21.946630 (+ 2776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:21.947279 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:21.947286 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:21.947288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:21.947416 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:21.947781 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:21.947788 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:21.947791 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:21.948132 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:21.948461 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:21.948687 (+ 226us) server_negotiation.cc:299] Negotiation successful
0924 16:15:21.948936 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":314,"threads_started":1}
I20260924 16:15:21.950628 11253 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:15:22.036509 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 11127
I20260924 16:15:22.052924 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 10726
I20260924 16:15:22.054253 10789 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:15:22.160482 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 10726
I20260924 16:15:22.183005 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:34419
--webserver_interface=127.29.234.62
--webserver_port=40281
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab
--principal=kudu/127.29.234.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:33053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.234.62:34419
--ranger_config_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.234.20:50657/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.29.234.20:50657/kms with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:15:22.516489 11270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:15:22.516863 11270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:15:22.517022 11270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:15:22.524510 11270 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:15:22.524662 11270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:15:22.524761 11270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:15:22.524873 11270 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:15:22.524955 11270 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:15:22.535466 11270 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:33053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.29.234.20:50657/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.29.234.62:34419
--ranger_config_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab
--principal=kudu/127.29.234.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:34419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.234.62
--webserver_port=40281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:15:22.536809 11270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:15:22.538190 11270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:15:22.550532 11276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:22.550597 11275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:22.550513 11278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:15:22.556952 11270 server_base.cc:1034] running on GCE node
I20260924 16:15:22.557896 11270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:15:22.559504 11270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:15:22.560743 11270 hybrid_clock.cc:648] HybridClock initialized: now 1790266522560682 us; error 71 us; skew 500 ppm
I20260924 16:15:22.561122 11270 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:15:22 dist-test-slave-v5j9 krb5kdc[10274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266522, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:15:22.564483 11270 init.cc:377] Logged in from keytab as kudu/127.29.234.62@KRBTEST.COM (short username kudu)
I20260924 16:15:22.566262 11270 webserver.cc:460] Webserver started at http://127.29.234.62:40281/ using document root <none> and password file <none>
I20260924 16:15:22.566970 11270 fs_manager.cc:359] Metadata directory not provided
I20260924 16:15:22.567121 11270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 16:15:22 dist-test-slave-v5j9 krb5kdc[10274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266522, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.62@KRBTEST.COM for HTTP/127.29.234.20@KRBTEST.COM
I20260924 16:15:22.617379 11270 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 16:15:22.621241 11286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:22.622977 11270 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:15:22.623173 11270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/master-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/master-0/wal
uuid: "10330b3a301d4583b27b45d298eddf4e"
format_stamp: "Formatted at 2026-09-24 16:15:18 on dist-test-slave-v5j9"
server_key: "73c7ac072fc81c2ffcda0fe9910c9392"
server_key_iv: "83700b1193cbb7d49192113f29c4056a"
server_key_version: "kuduclusterkey@0"
I20260924 16:15:22.624524 11270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:15:22.652763 11270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:15:22.656409 11270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:15:22.656656 11270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:15:22.675232 11270 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.62:34419
I20260924 16:15:22.675246 11338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.62:34419 every 8 connection(s)
I20260924 16:15:22.676743 11270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
I20260924 16:15:22.683935 11339 sys_catalog.cc:263] Verifying existing consensus state
I20260924 16:15:22.687254 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 11270
I20260924 16:15:22.687678 11339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10330b3a301d4583b27b45d298eddf4e: Bootstrap starting.
I20260924 16:15:22.688355 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.234.20:50657/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.1:35129
--local_ip_for_outbound_sockets=127.29.234.1
--tserver_master_addrs=127.29.234.62:34419
--webserver_port=39285
--webserver_interface=127.29.234.1
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab
--principal=kudu/127.29.234.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:33053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.29.234.20:50657/kms with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:15:22.709532 11339 log.cc:824] T 00000000000000000000000000000000 P 10330b3a301d4583b27b45d298eddf4e: Log is configured to *not* fsync() on all Append() calls
I20260924 16:15:22.718884 11339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10330b3a301d4583b27b45d298eddf4e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 16:15:22.719430 11339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10330b3a301d4583b27b45d298eddf4e: Bootstrap complete.
I20260924 16:15:22.728432 11339 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10330b3a301d4583b27b45d298eddf4e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10330b3a301d4583b27b45d298eddf4e" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 34419 } }
I20260924 16:15:22.729424 11339 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10330b3a301d4583b27b45d298eddf4e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10330b3a301d4583b27b45d298eddf4e, State: Initialized, Role: FOLLOWER
I20260924 16:15:22.730151 11339 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10330b3a301d4583b27b45d298eddf4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10330b3a301d4583b27b45d298eddf4e" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 34419 } }
I20260924 16:15:22.730412 11339 raft_consensus.cc:388] T 00000000000000000000000000000000 P 10330b3a301d4583b27b45d298eddf4e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:15:22.730549 11339 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10330b3a301d4583b27b45d298eddf4e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:15:22.730741 11339 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10330b3a301d4583b27b45d298eddf4e [term 1 FOLLOWER]: Advancing to term 2
I20260924 16:15:22.734611 11339 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10330b3a301d4583b27b45d298eddf4e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10330b3a301d4583b27b45d298eddf4e" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 34419 } }
I20260924 16:15:22.735101 11339 leader_election.cc:302] T 00000000000000000000000000000000 P 10330b3a301d4583b27b45d298eddf4e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 10330b3a301d4583b27b45d298eddf4e; no voters:
I20260924 16:15:22.735733 11339 leader_election.cc:288] T 00000000000000000000000000000000 P 10330b3a301d4583b27b45d298eddf4e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 16:15:22.735868 11343 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10330b3a301d4583b27b45d298eddf4e [term 2 FOLLOWER]: Leader election won for term 2
I20260924 16:15:22.737044 11343 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10330b3a301d4583b27b45d298eddf4e [term 2 LEADER]: Becoming Leader. State: Replica: 10330b3a301d4583b27b45d298eddf4e, State: Running, Role: LEADER
I20260924 16:15:22.737656 11343 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10330b3a301d4583b27b45d298eddf4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10330b3a301d4583b27b45d298eddf4e" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 34419 } }
I20260924 16:15:22.737952 11339 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10330b3a301d4583b27b45d298eddf4e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:15:22.740305 11345 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10330b3a301d4583b27b45d298eddf4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10330b3a301d4583b27b45d298eddf4e. Latest consensus state: current_term: 2 leader_uuid: "10330b3a301d4583b27b45d298eddf4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10330b3a301d4583b27b45d298eddf4e" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 34419 } } }
I20260924 16:15:22.740393 11344 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10330b3a301d4583b27b45d298eddf4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "10330b3a301d4583b27b45d298eddf4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10330b3a301d4583b27b45d298eddf4e" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 34419 } } }
I20260924 16:15:22.740739 11345 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10330b3a301d4583b27b45d298eddf4e [sys.catalog]: This master's current role is: LEADER
I20260924 16:15:22.740820 11344 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10330b3a301d4583b27b45d298eddf4e [sys.catalog]: This master's current role is: LEADER
I20260924 16:15:22.741741 11348 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:15:22.758432 11348 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:15:22.760324 11348 catalog_manager.cc:1259] Loaded cluster ID: 0e72f217e0ea4dff97106ed2004bc219
I20260924 16:15:22.760524 11348 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:15:22.765754 11348 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:15:22.767901 11348 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 10330b3a301d4583b27b45d298eddf4e: Loaded TSK: 0
I20260924 16:15:22.770084 11348 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:15:22.776686 11339 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260924 16:15:23.166620 11341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:15:23.167339 11341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:15:23.167547 11341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:15:23.180143 11341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:15:23.180280 11341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:15:23.180454 11341 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.1
I20260924 16:15:23.196412 11341 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:33053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.29.234.20:50657/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab
--principal=kudu/127.29.234.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.1:35129
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.234.1
--webserver_port=39285
--tserver_master_addrs=127.29.234.62:34419
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.1
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:15:23.198406 11341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:15:23.200186 11341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:15:23.226809 11377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:23.233861 11376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:15:23.266600 11341 server_base.cc:1034] running on GCE node
W20260924 16:15:23.267704 11379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:15:23.268682 11341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:15:23.270262 11341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:15:23.271610 11341 hybrid_clock.cc:648] HybridClock initialized: now 1790266523271517 us; error 76 us; skew 500 ppm
I20260924 16:15:23.272089 11341 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:15:23 dist-test-slave-v5j9 krb5kdc[10274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266523, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:15:23.276620 11341 init.cc:377] Logged in from keytab as kudu/127.29.234.1@KRBTEST.COM (short username kudu)
I20260924 16:15:23.278650 11341 webserver.cc:460] Webserver started at http://127.29.234.1:39285/ using document root <none> and password file <none>
I20260924 16:15:23.279742 11341 fs_manager.cc:359] Metadata directory not provided
I20260924 16:15:23.279959 11341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 16:15:23 dist-test-slave-v5j9 krb5kdc[10274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266523, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.1@KRBTEST.COM for HTTP/127.29.234.20@KRBTEST.COM
I20260924 16:15:23.312376 11341 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 16:15:23.316047 11387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:23.317888 11341 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260924 16:15:23.318081 11341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-0/wal
uuid: "c5d39193651b4ebeb5b0ee4a56f32173"
format_stamp: "Formatted at 2026-09-24 16:15:19 on dist-test-slave-v5j9"
server_key: "c778fc799ce0da748a0b759290e0e7d4"
server_key_iv: "55abf7b230bdf2420a8e12b34d3873a5"
server_key_version: "kuduclusterkey@0"
I20260924 16:15:23.319581 11341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:15:23.365617 11341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:15:23.370790 11341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:15:23.371107 11341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:15:23.372382 11341 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:15:23.374280 11341 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:15:23.374385 11341 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:23.374488 11341 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:15:23.374558 11341 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:23.438511 11341 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.1:35129
I20260924 16:15:23.438553 11503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.1:35129 every 8 connection(s)
I20260924 16:15:23.440263 11341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
I20260924 16:15:23.441694 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 11341
I20260924 16:15:23.442598 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.234.20:50657/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.2:36011
--local_ip_for_outbound_sockets=127.29.234.2
--tserver_master_addrs=127.29.234.62:34419
--webserver_port=42339
--webserver_interface=127.29.234.2
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab
--principal=kudu/127.29.234.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:33053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.29.234.20:50657/kms with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 16:15:23 dist-test-slave-v5j9 krb5kdc[10274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266523, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.1@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:15:23.483073 11506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:23.444557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:34419 (local address 127.29.234.1:34339)
0924 16:15:23.450559 (+ 6002us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:23.450613 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:23.452743 (+ 2130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:23.454644 (+ 1901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:23.454654 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:23.455215 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:15:23.455766 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:23.455783 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:23.457390 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:23.457397 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:23.458643 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:23.458663 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:23.460489 (+ 1826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:23.460497 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:23.460790 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:23.468461 (+ 7671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:15:23.468509 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:15:23.472227 (+ 3718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:15:23.477516 (+ 5289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:15:23.477528 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:15:23.477548 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:15:23.478215 (+ 667us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:15:23.478846 (+ 631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:15:23.478851 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:15:23.478855 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:15:23.479095 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:15:23.480263 (+ 1168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:15:23.480273 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:15:23.480684 (+ 411us) client_negotiation.cc:769] Sending connection context
0924 16:15:23.480999 (+ 315us) client_negotiation.cc:240] Negotiation successful
0924 16:15:23.481698 (+ 699us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5708,"thread_start_us":5465,"threads_started":1}
I20260924 16:15:23.483125 11507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:23.445929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.1:34339 (local address 127.29.234.62:34419)
0924 16:15:23.449936 (+ 4007us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:23.449963 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:23.451072 (+ 1109us) server_negotiation.cc:407] Connection header received
0924 16:15:23.453592 (+ 2520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:23.453614 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:23.454108 (+ 494us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:23.454503 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:23.455978 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:23.457204 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:23.458806 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:23.459920 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:23.460109 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:23.472682 (+ 12573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:23.472698 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:23.472728 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:15:23.472808 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:23.477281 (+ 4473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:23.478431 (+ 1150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:23.478442 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:23.478452 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:23.478691 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:23.479246 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:23.479252 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:23.479255 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:23.480095 (+ 840us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:23.480314 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:23.481303 (+ 989us) server_negotiation.cc:299] Negotiation successful
0924 16:15:23.481818 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3703,"thread_start_us":3463,"threads_started":1}
I20260924 16:15:23.487133 11504 heartbeater.cc:347] Connected to a master server at 127.29.234.62:34419
I20260924 16:15:23.487460 11504 heartbeater.cc:464] Registering TS with master...
I20260924 16:15:23.488332 11504 heartbeater.cc:511] Master 127.29.234.62:34419 requested a full tablet report, sending...
W20260924 16:15:23.491978 11504 heartbeater.cc:644] Failed to heartbeat to 127.29.234.62:34419 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 16:15:23.492219 11504 heartbeater.cc:464] Registering TS with master...
I20260924 16:15:23.492550 11504 heartbeater.cc:511] Master 127.29.234.62:34419 requested a full tablet report, sending...
I20260924 16:15:23.493952 11504 heartbeater.cc:464] Registering TS with master...
I20260924 16:15:23.494325 11504 heartbeater.cc:511] Master 127.29.234.62:34419 requested a full tablet report, sending...
W20260924 16:15:23.495572 11504 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 16:15:23.867303 11508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:15:23.867607 11508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:15:23.867785 11508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:15:23.875495 11508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:15:23.875645 11508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:15:23.875854 11508 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.2
I20260924 16:15:23.886720 11508 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:33053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.29.234.20:50657/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab
--principal=kudu/127.29.234.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.2:36011
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.234.2
--webserver_port=42339
--tserver_master_addrs=127.29.234.62:34419
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.2
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:15:23.888055 11508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:15:23.889531 11508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:15:23.906236 11523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:23.906539 11525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:23.907197 11522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:15:23.908109 11508 server_base.cc:1034] running on GCE node
I20260924 16:15:23.908977 11508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:15:23.910323 11508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:15:23.911607 11508 hybrid_clock.cc:648] HybridClock initialized: now 1790266523911507 us; error 97 us; skew 500 ppm
I20260924 16:15:23.912034 11508 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:15:23 dist-test-slave-v5j9 krb5kdc[10274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266523, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:15:23.916507 11508 init.cc:377] Logged in from keytab as kudu/127.29.234.2@KRBTEST.COM (short username kudu)
I20260924 16:15:23.918587 11508 webserver.cc:460] Webserver started at http://127.29.234.2:42339/ using document root <none> and password file <none>
I20260924 16:15:23.919613 11508 fs_manager.cc:359] Metadata directory not provided
I20260924 16:15:23.919775 11508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 16:15:23 dist-test-slave-v5j9 krb5kdc[10274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266523, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.2@KRBTEST.COM for HTTP/127.29.234.20@KRBTEST.COM
I20260924 16:15:23.955166 11508 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.000s sys 0.006s
I20260924 16:15:23.964784 11536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:23.966814 11508 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.002s sys 0.006s
I20260924 16:15:23.967087 11508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-1/wal
uuid: "0190b44255744fdd99cfb02c0951ec26"
format_stamp: "Formatted at 2026-09-24 16:15:20 on dist-test-slave-v5j9"
server_key: "b8698bb8b7ae509e9730dd8619ece987"
server_key_iv: "2e7665e30623e2a074ea7bf867d4701e"
server_key_version: "kuduclusterkey@0"
I20260924 16:15:23.968870 11508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:15:24.003105 11508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:15:24.008519 11508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:15:24.008823 11508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:15:24.010094 11508 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:15:24.011861 11508 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:15:24.012025 11508 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:24.012200 11508 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:15:24.012285 11508 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 16:15:24 dist-test-slave-v5j9 krb5kdc[10274](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790266524, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:15:24.089109 11508 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.2:36011
I20260924 16:15:24.089205 11649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.2:36011 every 8 connection(s)
I20260924 16:15:24.091130 11508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
I20260924 16:15:24.094767 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 11508
I20260924 16:15:24.095574 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.234.20:50657/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.3:35745
--local_ip_for_outbound_sockets=127.29.234.3
--tserver_master_addrs=127.29.234.62:34419
--webserver_port=37957
--webserver_interface=127.29.234.3
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab
--principal=kudu/127.29.234.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:33053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.29.234.20:50657/kms with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 16:15:24 dist-test-slave-v5j9 krb5kdc[10274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266523, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.2@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:15:24.129966 11653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:24.095414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:34419 (local address 127.29.234.2:39483)
0924 16:15:24.105545 (+ 10131us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:24.105627 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:24.108551 (+ 2924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:24.109416 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:24.109432 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:24.110123 (+ 691us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:15:24.111005 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:24.111025 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:24.112679 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:24.112687 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:24.114850 (+ 2163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:24.114877 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:24.116302 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:24.116311 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:24.116628 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:24.117453 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:15:24.117486 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:15:24.120464 (+ 2978us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:15:24.124817 (+ 4353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:15:24.124827 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:15:24.124846 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:15:24.125498 (+ 652us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:15:24.126027 (+ 529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:15:24.126032 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:15:24.126036 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:15:24.126305 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:15:24.126979 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:15:24.126988 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:15:24.127450 (+ 462us) client_negotiation.cc:769] Sending connection context
0924 16:15:24.127739 (+ 289us) client_negotiation.cc:240] Negotiation successful
0924 16:15:24.128394 (+ 655us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9628,"thread_start_us":9371,"threads_started":1}
I20260924 16:15:24.131903 11652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:24.094668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.2:39483 (local address 127.29.234.62:34419)
0924 16:15:24.107258 (+ 12590us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:24.107264 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:24.107289 (+ 25us) server_negotiation.cc:407] Connection header received
0924 16:15:24.108858 (+ 1569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:24.108863 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:24.108961 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:24.109093 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:24.111197 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:24.112465 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:24.115028 (+ 2563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:24.115829 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:24.115939 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:24.120790 (+ 4851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:24.120797 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:24.120803 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:15:24.120847 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:24.124645 (+ 3798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:24.125676 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:24.125723 (+ 47us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:24.125729 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:24.125889 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:24.126461 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:24.126467 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:24.126470 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:24.126825 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:24.131171 (+ 4346us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:24.131307 (+ 136us) server_negotiation.cc:299] Negotiation successful
0924 16:15:24.131546 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12445,"thread_start_us":12617,"threads_started":1}
I20260924 16:15:24.133663 11650 heartbeater.cc:347] Connected to a master server at 127.29.234.62:34419
I20260924 16:15:24.134001 11650 heartbeater.cc:464] Registering TS with master...
I20260924 16:15:24.134914 11650 heartbeater.cc:511] Master 127.29.234.62:34419 requested a full tablet report, sending...
I20260924 16:15:24.137148 11304 ts_manager.cc:194] Registered new tserver with Master: 0190b44255744fdd99cfb02c0951ec26 (127.29.234.2:36011)
I20260924 16:15:24.139374 11304 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.234.2@KRBTEST.COM'} at 127.29.234.2:39483
Sep 24 16:15:24 dist-test-slave-v5j9 krb5kdc[10274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266524, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.62@KRBTEST.COM for HTTP/127.29.234.20@KRBTEST.COM
I20260924 16:15:24.495718 11504 heartbeater.cc:464] Registering TS with master...
I20260924 16:15:24.496201 11504 heartbeater.cc:511] Master 127.29.234.62:34419 requested a full tablet report, sending...
I20260924 16:15:24.497562 11304 ts_manager.cc:194] Registered new tserver with Master: c5d39193651b4ebeb5b0ee4a56f32173 (127.29.234.1:35129)
I20260924 16:15:24.498747 11304 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.234.1@KRBTEST.COM'} at 127.29.234.1:34339
W20260924 16:15:24.583592 11654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:15:24.584009 11654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:15:24.584148 11654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:15:24.595255 11654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:15:24.595391 11654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:15:24.595618 11654 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.3
I20260924 16:15:24.610742 11654 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:33053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.29.234.20:50657/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab
--principal=kudu/127.29.234.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.3:35745
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.234.3
--webserver_port=37957
--tserver_master_addrs=127.29.234.62:34419
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.3
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:15:24.612404 11654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:15:24.614073 11654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:15:24.630760 11662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:24.630267 11661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:24.632827 11664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:15:24.633965 11654 server_base.cc:1034] running on GCE node
I20260924 16:15:24.634676 11654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:15:24.635828 11654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:15:24.637128 11654 hybrid_clock.cc:648] HybridClock initialized: now 1790266524637052 us; error 57 us; skew 500 ppm
I20260924 16:15:24.637550 11654 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:15:24 dist-test-slave-v5j9 krb5kdc[10274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266524, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:15:24.641592 11654 init.cc:377] Logged in from keytab as kudu/127.29.234.3@KRBTEST.COM (short username kudu)
I20260924 16:15:24.650916 11654 webserver.cc:460] Webserver started at http://127.29.234.3:37957/ using document root <none> and password file <none>
I20260924 16:15:24.651885 11654 fs_manager.cc:359] Metadata directory not provided
I20260924 16:15:24.652040 11654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 16:15:24 dist-test-slave-v5j9 krb5kdc[10274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266524, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.3@KRBTEST.COM for HTTP/127.29.234.20@KRBTEST.COM
I20260924 16:15:24.680933 11654 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 16:15:24.685477 11672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:24.687127 11654 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260924 16:15:24.687341 11654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-2/wal
uuid: "6712eda39e82456db9a5d7e53c096263"
format_stamp: "Formatted at 2026-09-24 16:15:21 on dist-test-slave-v5j9"
server_key: "c4a619c430fb7a5115a626e31fea1622"
server_key_iv: "55fdb6fed2ebd721c917562af38fc6a0"
server_key_version: "kuduclusterkey@0"
I20260924 16:15:24.688431 11654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:15:24.725368 11654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:15:24.731179 11654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:15:24.731551 11654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:15:24.732897 11654 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:15:24.734846 11654 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:15:24.735008 11654 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:24.735153 11654 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:15:24.735263 11654 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 16:15:24 dist-test-slave-v5j9 krb5kdc[10274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266524, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.62@KRBTEST.COM for HTTP/127.29.234.20@KRBTEST.COM
I20260924 16:15:24.978092 11654 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.3:35745
I20260924 16:15:24.980830 11786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.3:35745 every 8 connection(s)
I20260924 16:15:24.981083 11654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
I20260924 16:15:24.989406 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 11654
I20260924 16:15:25.039196 11790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:24.990709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50400 (local address 127.29.234.62:34419)
0924 16:15:24.995553 (+ 4844us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:24.995564 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:25.018045 (+ 22481us) server_negotiation.cc:407] Connection header received
0924 16:15:25.018296 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:25.018302 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:25.018398 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:25.018533 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:25.020533 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:25.021309 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:25.023432 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:25.024055 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:25.024198 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:25.027998 (+ 3800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:25.028004 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:25.028007 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:15:25.028076 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:25.031931 (+ 3855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:25.033374 (+ 1443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:25.033381 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:25.033387 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:25.033561 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:25.036142 (+ 2581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:25.036147 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:25.036151 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:25.036493 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:25.036767 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:25.038412 (+ 1645us) server_negotiation.cc:299] Negotiation successful
0924 16:15:25.038682 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4666,"thread_start_us":4523,"threads_started":1}
Sep 24 16:15:25 dist-test-slave-v5j9 krb5kdc[10274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266524, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.3@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:15:25.048311 11792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:24.997222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.3:35347 (local address 127.29.234.62:34419)
0924 16:15:25.022956 (+ 25734us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:25.022962 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:25.024940 (+ 1978us) server_negotiation.cc:407] Connection header received
0924 16:15:25.028131 (+ 3191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:25.028138 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:25.028237 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:25.028358 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:25.031701 (+ 3343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:25.032684 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:25.035079 (+ 2395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:25.035755 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:25.035908 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:25.041923 (+ 6015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:25.041931 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:25.041938 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:15:25.041982 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:25.045097 (+ 3115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:25.045936 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:25.045941 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:25.045945 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:25.046074 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:25.046525 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:25.046530 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:25.046533 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:25.046812 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:25.046979 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:25.047716 (+ 737us) server_negotiation.cc:299] Negotiation successful
0924 16:15:25.047925 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25585,"thread_start_us":25748,"threads_started":1}
I20260924 16:15:25.049476 11791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:24.996445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:34419 (local address 127.29.234.3:35347)
0924 16:15:25.023769 (+ 27324us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:25.023854 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:25.027750 (+ 3896us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:25.029277 (+ 1527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:25.029292 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:25.030429 (+ 1137us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:15:25.031436 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:25.031472 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:25.033530 (+ 2058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:25.033535 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:25.034913 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:25.034933 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:25.036025 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:25.036034 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:25.036355 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:25.038504 (+ 2149us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:15:25.038542 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:15:25.041584 (+ 3042us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:15:25.045251 (+ 3667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:15:25.045262 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:15:25.045281 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:15:25.045801 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:15:25.046196 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:15:25.046201 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:15:25.046205 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:15:25.046403 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:15:25.046946 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:15:25.046957 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:15:25.047339 (+ 382us) client_negotiation.cc:769] Sending connection context
0924 16:15:25.047613 (+ 274us) client_negotiation.cc:240] Negotiation successful
0924 16:15:25.048182 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26861,"thread_start_us":28351,"threads_started":1}
I20260924 16:15:25.051084 11787 heartbeater.cc:347] Connected to a master server at 127.29.234.62:34419
I20260924 16:15:25.051414 11787 heartbeater.cc:464] Registering TS with master...
I20260924 16:15:25.052160 11787 heartbeater.cc:511] Master 127.29.234.62:34419 requested a full tablet report, sending...
I20260924 16:15:25.053737 11304 ts_manager.cc:194] Registered new tserver with Master: 6712eda39e82456db9a5d7e53c096263 (127.29.234.3:35745)
I20260924 16:15:25.054888 11304 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.234.3@KRBTEST.COM'} at 127.29.234.3:35347
I20260924 16:15:25.055817 30632 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:15:25.084657 11792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:25.063767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50414 (local address 127.29.234.62:34419)
0924 16:15:25.063999 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:25.064008 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:25.067697 (+ 3689us) server_negotiation.cc:407] Connection header received
0924 16:15:25.067922 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:25.067935 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:25.068001 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:25.068104 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:25.069942 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:25.071026 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:25.073168 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:25.073876 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:25.073984 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:25.077575 (+ 3591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:25.077583 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:25.077588 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:15:25.077629 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:25.081506 (+ 3877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:25.082399 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:25.082404 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:25.082408 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:25.082571 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:25.083048 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:25.083054 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:25.083059 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:25.083437 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:25.083712 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:25.084069 (+ 357us) server_negotiation.cc:299] Negotiation successful
0924 16:15:25.084296 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 16:15:25.095754 11303 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50414:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:15:25.107277 11303 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:15:25.133816 11810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:25.123319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46364 (local address 127.29.234.1:35129)
0924 16:15:25.123998 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:25.124007 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:25.124035 (+ 28us) server_negotiation.cc:407] Connection header received
0924 16:15:25.124271 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:25.124281 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:25.124537 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:25.124767 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:15:25.125928 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:25.127644 (+ 1716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:25.130800 (+ 3156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:25.132212 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:25.132348 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:25.132585 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:25.133232 (+ 647us) server_negotiation.cc:299] Negotiation successful
0924 16:15:25.133446 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":321,"threads_started":1}
I20260924 16:15:25.134132 11812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:25.124482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37086 (local address 127.29.234.3:35745)
0924 16:15:25.125633 (+ 1151us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:25.125644 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:25.125681 (+ 37us) server_negotiation.cc:407] Connection header received
0924 16:15:25.125783 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:25.125790 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:25.126172 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:25.127045 (+ 873us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:15:25.127306 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:25.128788 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:25.131407 (+ 2619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:25.132579 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:25.132721 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:25.133527 (+ 806us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:25.133611 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 16:15:25.133771 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":979,"thread_start_us":750,"threads_started":1}
I20260924 16:15:25.134696 11813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:25.125224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55688 (local address 127.29.234.2:36011)
0924 16:15:25.125908 (+ 684us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:25.125917 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:25.126467 (+ 550us) server_negotiation.cc:407] Connection header received
0924 16:15:25.126767 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:25.126775 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:25.126965 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:25.127122 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:15:25.127975 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:25.129158 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:25.131725 (+ 2567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:25.133235 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:25.133397 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:25.133733 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:25.134267 (+ 534us) server_negotiation.cc:299] Negotiation successful
0924 16:15:25.134465 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":462,"threads_started":1}
I20260924 16:15:25.134725 11811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:25.124222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.3:35745 (local address 127.0.0.1:37086)
0924 16:15:25.124788 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:25.124806 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:25.124976 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:25.126390 (+ 1414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:25.126394 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:25.126422 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:15:25.126831 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:25.126846 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:25.128929 (+ 2083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:25.128935 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:25.131238 (+ 2303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:25.131255 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:25.133092 (+ 1837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:25.133099 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:25.133267 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:25.133320 (+ 53us) client_negotiation.cc:769] Sending connection context
0924 16:15:25.133626 (+ 306us) client_negotiation.cc:240] Negotiation successful
0924 16:15:25.133778 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"mutex_wait_us":50,"thread_start_us":291,"threads_started":1}
I20260924 16:15:25.134757 11814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:25.125569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.2:36011 (local address 127.0.0.1:55688)
0924 16:15:25.126356 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:25.126374 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:25.126580 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:25.127394 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:25.127398 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:25.127421 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:15:25.127785 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:25.127797 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:25.129289 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:25.129303 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:25.131556 (+ 2253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:25.131572 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:25.133882 (+ 2310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:25.133890 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:25.134073 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:25.134101 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 16:15:25.134205 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 16:15:25.134346 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"mutex_wait_us":160,"thread_start_us":503,"threads_started":1}
I20260924 16:15:25.135310 11809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:25.122988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:35129 (local address 127.0.0.1:46364)
0924 16:15:25.123795 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:25.123840 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:25.124023 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:25.125165 (+ 1142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:25.125177 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:25.125215 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:15:25.125700 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:25.125715 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:25.127831 (+ 2116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:25.127840 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:25.130641 (+ 2801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:25.130658 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:25.132455 (+ 1797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:25.132462 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:25.132657 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:25.132727 (+ 70us) client_negotiation.cc:769] Sending connection context
0924 16:15:25.132851 (+ 124us) client_negotiation.cc:240] Negotiation successful
0924 16:15:25.133037 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"mutex_wait_us":31,"thread_start_us":326,"threads_started":1}
I20260924 16:15:25.139019 11584 tablet_service.cc:1467] Processing CreateTablet for tablet d2e72fa7decb4bf89ea1840fbfa543bc (DEFAULT_TABLE table=test-table [id=47a3b02fc56842dc8faa6e20e333ae90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:15:25.139019 11721 tablet_service.cc:1467] Processing CreateTablet for tablet d2e72fa7decb4bf89ea1840fbfa543bc (DEFAULT_TABLE table=test-table [id=47a3b02fc56842dc8faa6e20e333ae90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:15:25.139262 11437 tablet_service.cc:1467] Processing CreateTablet for tablet d2e72fa7decb4bf89ea1840fbfa543bc (DEFAULT_TABLE table=test-table [id=47a3b02fc56842dc8faa6e20e333ae90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:15:25.140511 11437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2e72fa7decb4bf89ea1840fbfa543bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:15:25.140514 11721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2e72fa7decb4bf89ea1840fbfa543bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:15:25.140538 11584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2e72fa7decb4bf89ea1840fbfa543bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:15:25.144917 11650 heartbeater.cc:503] Master 127.29.234.62:34419 was elected leader, sending a full tablet report...
I20260924 16:15:25.153056 11815 tablet_bootstrap.cc:492] T d2e72fa7decb4bf89ea1840fbfa543bc P c5d39193651b4ebeb5b0ee4a56f32173: Bootstrap starting.
I20260924 16:15:25.156984 11815 tablet_bootstrap.cc:654] T d2e72fa7decb4bf89ea1840fbfa543bc P c5d39193651b4ebeb5b0ee4a56f32173: Neither blocks nor log segments found. Creating new log.
I20260924 16:15:25.158063 11815 log.cc:824] T d2e72fa7decb4bf89ea1840fbfa543bc P c5d39193651b4ebeb5b0ee4a56f32173: Log is configured to *not* fsync() on all Append() calls
I20260924 16:15:25.160833 11815 tablet_bootstrap.cc:492] T d2e72fa7decb4bf89ea1840fbfa543bc P c5d39193651b4ebeb5b0ee4a56f32173: No bootstrap required, opened a new log
I20260924 16:15:25.161087 11815 ts_tablet_manager.cc:1397] T d2e72fa7decb4bf89ea1840fbfa543bc P c5d39193651b4ebeb5b0ee4a56f32173: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260924 16:15:25.162084 11817 tablet_bootstrap.cc:492] T d2e72fa7decb4bf89ea1840fbfa543bc P 6712eda39e82456db9a5d7e53c096263: Bootstrap starting.
I20260924 16:15:25.165118 11817 tablet_bootstrap.cc:654] T d2e72fa7decb4bf89ea1840fbfa543bc P 6712eda39e82456db9a5d7e53c096263: Neither blocks nor log segments found. Creating new log.
I20260924 16:15:25.165448 11816 tablet_bootstrap.cc:492] T d2e72fa7decb4bf89ea1840fbfa543bc P 0190b44255744fdd99cfb02c0951ec26: Bootstrap starting.
I20260924 16:15:25.166422 11817 log.cc:824] T d2e72fa7decb4bf89ea1840fbfa543bc P 6712eda39e82456db9a5d7e53c096263: Log is configured to *not* fsync() on all Append() calls
I20260924 16:15:25.168727 11816 tablet_bootstrap.cc:654] T d2e72fa7decb4bf89ea1840fbfa543bc P 0190b44255744fdd99cfb02c0951ec26: Neither blocks nor log segments found. Creating new log.
I20260924 16:15:25.168969 11815 raft_consensus.cc:348] T d2e72fa7decb4bf89ea1840fbfa543bc P c5d39193651b4ebeb5b0ee4a56f32173 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d39193651b4ebeb5b0ee4a56f32173" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 35129 } } peers { permanent_uuid: "6712eda39e82456db9a5d7e53c096263" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 35745 } } peers { permanent_uuid: "0190b44255744fdd99cfb02c0951ec26" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36011 } }
I20260924 16:15:25.169317 11815 raft_consensus.cc:374] T d2e72fa7decb4bf89ea1840fbfa543bc P c5d39193651b4ebeb5b0ee4a56f32173 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:15:25.169438 11815 raft_consensus.cc:729] T d2e72fa7decb4bf89ea1840fbfa543bc P c5d39193651b4ebeb5b0ee4a56f32173 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5d39193651b4ebeb5b0ee4a56f32173, State: Initialized, Role: FOLLOWER
I20260924 16:15:25.169550 11817 tablet_bootstrap.cc:492] T d2e72fa7decb4bf89ea1840fbfa543bc P 6712eda39e82456db9a5d7e53c096263: No bootstrap required, opened a new log
I20260924 16:15:25.169852 11817 ts_tablet_manager.cc:1397] T d2e72fa7decb4bf89ea1840fbfa543bc P 6712eda39e82456db9a5d7e53c096263: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260924 16:15:25.169961 11816 log.cc:824] T d2e72fa7decb4bf89ea1840fbfa543bc P 0190b44255744fdd99cfb02c0951ec26: Log is configured to *not* fsync() on all Append() calls
I20260924 16:15:25.170073 11815 consensus_queue.cc:260] T d2e72fa7decb4bf89ea1840fbfa543bc P c5d39193651b4ebeb5b0ee4a56f32173 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d39193651b4ebeb5b0ee4a56f32173" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 35129 } } peers { permanent_uuid: "6712eda39e82456db9a5d7e53c096263" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 35745 } } peers { permanent_uuid: "0190b44255744fdd99cfb02c0951ec26" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36011 } }
I20260924 16:15:25.171351 11504 heartbeater.cc:503] Master 127.29.234.62:34419 was elected leader, sending a full tablet report...
I20260924 16:15:25.171998 11815 ts_tablet_manager.cc:1428] T d2e72fa7decb4bf89ea1840fbfa543bc P c5d39193651b4ebeb5b0ee4a56f32173: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260924 16:15:25.173023 11816 tablet_bootstrap.cc:492] T d2e72fa7decb4bf89ea1840fbfa543bc P 0190b44255744fdd99cfb02c0951ec26: No bootstrap required, opened a new log
I20260924 16:15:25.173311 11816 ts_tablet_manager.cc:1397] T d2e72fa7decb4bf89ea1840fbfa543bc P 0190b44255744fdd99cfb02c0951ec26: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.000s
I20260924 16:15:25.179665 11817 raft_consensus.cc:348] T d2e72fa7decb4bf89ea1840fbfa543bc P 6712eda39e82456db9a5d7e53c096263 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d39193651b4ebeb5b0ee4a56f32173" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 35129 } } peers { permanent_uuid: "6712eda39e82456db9a5d7e53c096263" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 35745 } } peers { permanent_uuid: "0190b44255744fdd99cfb02c0951ec26" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36011 } }
I20260924 16:15:25.180013 11817 raft_consensus.cc:374] T d2e72fa7decb4bf89ea1840fbfa543bc P 6712eda39e82456db9a5d7e53c096263 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:15:25.180136 11817 raft_consensus.cc:729] T d2e72fa7decb4bf89ea1840fbfa543bc P 6712eda39e82456db9a5d7e53c096263 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6712eda39e82456db9a5d7e53c096263, State: Initialized, Role: FOLLOWER
I20260924 16:15:25.180673 11817 consensus_queue.cc:260] T d2e72fa7decb4bf89ea1840fbfa543bc P 6712eda39e82456db9a5d7e53c096263 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d39193651b4ebeb5b0ee4a56f32173" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 35129 } } peers { permanent_uuid: "6712eda39e82456db9a5d7e53c096263" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 35745 } } peers { permanent_uuid: "0190b44255744fdd99cfb02c0951ec26" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36011 } }
I20260924 16:15:25.181768 11787 heartbeater.cc:503] Master 127.29.234.62:34419 was elected leader, sending a full tablet report...
I20260924 16:15:25.182248 11817 ts_tablet_manager.cc:1428] T d2e72fa7decb4bf89ea1840fbfa543bc P 6712eda39e82456db9a5d7e53c096263: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260924 16:15:25.182401 11816 raft_consensus.cc:348] T d2e72fa7decb4bf89ea1840fbfa543bc P 0190b44255744fdd99cfb02c0951ec26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d39193651b4ebeb5b0ee4a56f32173" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 35129 } } peers { permanent_uuid: "6712eda39e82456db9a5d7e53c096263" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 35745 } } peers { permanent_uuid: "0190b44255744fdd99cfb02c0951ec26" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36011 } }
I20260924 16:15:25.182813 11816 raft_consensus.cc:374] T d2e72fa7decb4bf89ea1840fbfa543bc P 0190b44255744fdd99cfb02c0951ec26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:15:25.182924 11816 raft_consensus.cc:729] T d2e72fa7decb4bf89ea1840fbfa543bc P 0190b44255744fdd99cfb02c0951ec26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0190b44255744fdd99cfb02c0951ec26, State: Initialized, Role: FOLLOWER
I20260924 16:15:25.183494 11816 consensus_queue.cc:260] T d2e72fa7decb4bf89ea1840fbfa543bc P 0190b44255744fdd99cfb02c0951ec26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d39193651b4ebeb5b0ee4a56f32173" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 35129 } } peers { permanent_uuid: "6712eda39e82456db9a5d7e53c096263" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 35745 } } peers { permanent_uuid: "0190b44255744fdd99cfb02c0951ec26" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36011 } }
I20260924 16:15:25.185042 11816 ts_tablet_manager.cc:1428] T d2e72fa7decb4bf89ea1840fbfa543bc P 0190b44255744fdd99cfb02c0951ec26: Time spent starting tablet: real 0.012s user 0.010s sys 0.000s
W20260924 16:15:25.198928 11505 tablet.cc:2369] T d2e72fa7decb4bf89ea1840fbfa543bc P c5d39193651b4ebeb5b0ee4a56f32173: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 16:15:25.254499 11788 tablet.cc:2369] T d2e72fa7decb4bf89ea1840fbfa543bc P 6712eda39e82456db9a5d7e53c096263: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:15:25.276295 11822 raft_consensus.cc:482] T d2e72fa7decb4bf89ea1840fbfa543bc P 6712eda39e82456db9a5d7e53c096263 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:15:25.276630 11822 raft_consensus.cc:504] T d2e72fa7decb4bf89ea1840fbfa543bc P 6712eda39e82456db9a5d7e53c096263 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d39193651b4ebeb5b0ee4a56f32173" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 35129 } } peers { permanent_uuid: "6712eda39e82456db9a5d7e53c096263" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 35745 } } peers { permanent_uuid: "0190b44255744fdd99cfb02c0951ec26" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36011 } }
I20260924 16:15:25.278415 11822 leader_election.cc:288] T d2e72fa7decb4bf89ea1840fbfa543bc P 6712eda39e82456db9a5d7e53c096263 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5d39193651b4ebeb5b0ee4a56f32173 (127.29.234.1:35129), 0190b44255744fdd99cfb02c0951ec26 (127.29.234.2:36011)
I20260924 16:15:25.285640 11810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:25.278918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.3:44027 (local address 127.29.234.1:35129)
0924 16:15:25.279240 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:25.279248 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:25.279269 (+ 21us) server_negotiation.cc:407] Connection header received
0924 16:15:25.279350 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:25.279355 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:25.279429 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:25.279524 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:15:25.280083 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:25.281029 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:25.283527 (+ 2498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:25.284734 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:25.284899 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:25.284977 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:25.285242 (+ 265us) server_negotiation.cc:299] Negotiation successful
0924 16:15:25.285388 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260924 16:15:25.285645 11791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:25.278675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:35129 (local address 127.29.234.3:44027)
0924 16:15:25.278876 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:25.278899 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:25.279037 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:25.279673 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:25.279678 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:25.279707 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:15:25.279921 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:25.279934 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:25.281169 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:25.281174 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:25.283363 (+ 2189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:25.283377 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:25.284894 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:25.284898 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:25.285027 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:25.285086 (+ 59us) client_negotiation.cc:769] Sending connection context
0924 16:15:25.285192 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 16:15:25.285337 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64,"mutex_wait_us":71}
I20260924 16:15:25.285950 11813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:25.279147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.3:35537 (local address 127.29.234.2:36011)
0924 16:15:25.279358 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:25.279364 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:25.279547 (+ 183us) server_negotiation.cc:407] Connection header received
0924 16:15:25.279809 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:25.279813 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:25.279884 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:25.279980 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:15:25.280650 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:25.281443 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:25.283753 (+ 2310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:25.284976 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:25.285123 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:25.285206 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:25.285541 (+ 335us) server_negotiation.cc:299] Negotiation successful
0924 16:15:25.285691 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 16:15:25.286041 11824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:25.279013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.2:36011 (local address 127.29.234.3:35537)
0924 16:15:25.279462 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:25.279479 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:25.279636 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:25.280110 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:25.280116 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:25.280145 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:15:25.280483 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:25.280494 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:25.281600 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:25.281605 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:25.283594 (+ 1989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:25.283604 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:25.285173 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:25.285182 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:25.285388 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:25.285415 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 16:15:25.285500 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 16:15:25.285612 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"mutex_wait_us":1,"thread_start_us":440,"threads_started":1}
I20260924 16:15:25.286823 11459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2e72fa7decb4bf89ea1840fbfa543bc" candidate_uuid: "6712eda39e82456db9a5d7e53c096263" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5d39193651b4ebeb5b0ee4a56f32173" is_pre_election: true
I20260924 16:15:25.287148 11604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2e72fa7decb4bf89ea1840fbfa543bc" candidate_uuid: "6712eda39e82456db9a5d7e53c096263" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0190b44255744fdd99cfb02c0951ec26" is_pre_election: true
I20260924 16:15:25.287469 11459 raft_consensus.cc:2454] T d2e72fa7decb4bf89ea1840fbfa543bc P c5d39193651b4ebeb5b0ee4a56f32173 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6712eda39e82456db9a5d7e53c096263 in term 0.
I20260924 16:15:25.287703 11604 raft_consensus.cc:2454] T d2e72fa7decb4bf89ea1840fbfa543bc P 0190b44255744fdd99cfb02c0951ec26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6712eda39e82456db9a5d7e53c096263 in term 0.
I20260924 16:15:25.288255 11675 leader_election.cc:302] T d2e72fa7decb4bf89ea1840fbfa543bc P 6712eda39e82456db9a5d7e53c096263 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6712eda39e82456db9a5d7e53c096263, c5d39193651b4ebeb5b0ee4a56f32173; no voters:
I20260924 16:15:25.288635 11822 raft_consensus.cc:2781] T d2e72fa7decb4bf89ea1840fbfa543bc P 6712eda39e82456db9a5d7e53c096263 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:15:25.288757 11822 raft_consensus.cc:482] T d2e72fa7decb4bf89ea1840fbfa543bc P 6712eda39e82456db9a5d7e53c096263 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:15:25.288916 11822 raft_consensus.cc:3037] T d2e72fa7decb4bf89ea1840fbfa543bc P 6712eda39e82456db9a5d7e53c096263 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:15:25.291198 11822 raft_consensus.cc:504] T d2e72fa7decb4bf89ea1840fbfa543bc P 6712eda39e82456db9a5d7e53c096263 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d39193651b4ebeb5b0ee4a56f32173" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 35129 } } peers { permanent_uuid: "6712eda39e82456db9a5d7e53c096263" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 35745 } } peers { permanent_uuid: "0190b44255744fdd99cfb02c0951ec26" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36011 } }
I20260924 16:15:25.291769 11822 leader_election.cc:288] T d2e72fa7decb4bf89ea1840fbfa543bc P 6712eda39e82456db9a5d7e53c096263 [CANDIDATE]: Term 1 election: Requested vote from peers c5d39193651b4ebeb5b0ee4a56f32173 (127.29.234.1:35129), 0190b44255744fdd99cfb02c0951ec26 (127.29.234.2:36011)
I20260924 16:15:25.292333 11459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2e72fa7decb4bf89ea1840fbfa543bc" candidate_uuid: "6712eda39e82456db9a5d7e53c096263" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5d39193651b4ebeb5b0ee4a56f32173"
I20260924 16:15:25.292399 11604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2e72fa7decb4bf89ea1840fbfa543bc" candidate_uuid: "6712eda39e82456db9a5d7e53c096263" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0190b44255744fdd99cfb02c0951ec26"
I20260924 16:15:25.292536 11459 raft_consensus.cc:3037] T d2e72fa7decb4bf89ea1840fbfa543bc P c5d39193651b4ebeb5b0ee4a56f32173 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:15:25.292598 11604 raft_consensus.cc:3037] T d2e72fa7decb4bf89ea1840fbfa543bc P 0190b44255744fdd99cfb02c0951ec26 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:15:25.294847 11604 raft_consensus.cc:2454] T d2e72fa7decb4bf89ea1840fbfa543bc P 0190b44255744fdd99cfb02c0951ec26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6712eda39e82456db9a5d7e53c096263 in term 1.
I20260924 16:15:25.294965 11459 raft_consensus.cc:2454] T d2e72fa7decb4bf89ea1840fbfa543bc P c5d39193651b4ebeb5b0ee4a56f32173 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6712eda39e82456db9a5d7e53c096263 in term 1.
I20260924 16:15:25.295277 11675 leader_election.cc:302] T d2e72fa7decb4bf89ea1840fbfa543bc P 6712eda39e82456db9a5d7e53c096263 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0190b44255744fdd99cfb02c0951ec26, 6712eda39e82456db9a5d7e53c096263; no voters:
I20260924 16:15:25.295532 11822 raft_consensus.cc:2781] T d2e72fa7decb4bf89ea1840fbfa543bc P 6712eda39e82456db9a5d7e53c096263 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:15:25.295954 11822 raft_consensus.cc:686] T d2e72fa7decb4bf89ea1840fbfa543bc P 6712eda39e82456db9a5d7e53c096263 [term 1 LEADER]: Becoming Leader. State: Replica: 6712eda39e82456db9a5d7e53c096263, State: Running, Role: LEADER
I20260924 16:15:25.296415 11822 consensus_queue.cc:237] T d2e72fa7decb4bf89ea1840fbfa543bc P 6712eda39e82456db9a5d7e53c096263 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d39193651b4ebeb5b0ee4a56f32173" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 35129 } } peers { permanent_uuid: "6712eda39e82456db9a5d7e53c096263" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 35745 } } peers { permanent_uuid: "0190b44255744fdd99cfb02c0951ec26" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36011 } }
I20260924 16:15:25.301012 11303 catalog_manager.cc:5495] T d2e72fa7decb4bf89ea1840fbfa543bc P 6712eda39e82456db9a5d7e53c096263 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6712eda39e82456db9a5d7e53c096263 (127.29.234.3). New cstate: current_term: 1 leader_uuid: "6712eda39e82456db9a5d7e53c096263" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5d39193651b4ebeb5b0ee4a56f32173" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 35129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6712eda39e82456db9a5d7e53c096263" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 35745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0190b44255744fdd99cfb02c0951ec26" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36011 } health_report { overall_health: UNKNOWN } } }
I20260924 16:15:25.329108 11812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:25.323769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37100 (local address 127.29.234.3:35745)
0924 16:15:25.323971 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:25.323975 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:25.323992 (+ 17us) server_negotiation.cc:407] Connection header received
0924 16:15:25.324176 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:25.324180 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:25.324253 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:25.324378 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:15:25.324775 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:25.325584 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:25.327099 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:25.327465 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:25.327552 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:25.328012 (+ 460us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:15:25.328541 (+ 529us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:15:25.328658 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:25.328783 (+ 125us) server_negotiation.cc:299] Negotiation successful
0924 16:15:25.328843 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 16:15:25.337281 11604 raft_consensus.cc:1261] T d2e72fa7decb4bf89ea1840fbfa543bc P 0190b44255744fdd99cfb02c0951ec26 [term 1 FOLLOWER]: Refusing update from remote peer 6712eda39e82456db9a5d7e53c096263: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:15:25.337282 11459 raft_consensus.cc:1261] T d2e72fa7decb4bf89ea1840fbfa543bc P c5d39193651b4ebeb5b0ee4a56f32173 [term 1 FOLLOWER]: Refusing update from remote peer 6712eda39e82456db9a5d7e53c096263: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:15:25.338320 11825 consensus_queue.cc:1035] T d2e72fa7decb4bf89ea1840fbfa543bc P 6712eda39e82456db9a5d7e53c096263 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5d39193651b4ebeb5b0ee4a56f32173" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 35129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:15:25.338696 11822 consensus_queue.cc:1035] T d2e72fa7decb4bf89ea1840fbfa543bc P 6712eda39e82456db9a5d7e53c096263 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0190b44255744fdd99cfb02c0951ec26" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:15:25.352010 11829 mvcc.cc:205] Tried to move back new op lower bound from 7332931687771541504 to 7332931687618183168. Current Snapshot: MvccSnapshot[applied={T|T < 7332931687771541504}]
I20260924 16:15:25.352025 11830 mvcc.cc:205] Tried to move back new op lower bound from 7332931687771541504 to 7332931687618183168. Current Snapshot: MvccSnapshot[applied={T|T < 7332931687771541504}]
I20260924 16:15:25.364137 11303 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50414:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:15:25.364415 11303 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50414:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:15:25.368134 11303 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 10330b3a301d4583b27b45d298eddf4e: Sending DeleteTablet for 3 replicas of tablet d2e72fa7decb4bf89ea1840fbfa543bc
I20260924 16:15:25.369280 11721 tablet_service.cc:1514] Processing DeleteTablet for tablet d2e72fa7decb4bf89ea1840fbfa543bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:15:25 UTC) from {username='kudu', principal='kudu/127.29.234.62@KRBTEST.COM'} at 127.0.0.1:37086
I20260924 16:15:25.369273 11437 tablet_service.cc:1514] Processing DeleteTablet for tablet d2e72fa7decb4bf89ea1840fbfa543bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:15:25 UTC) from {username='kudu', principal='kudu/127.29.234.62@KRBTEST.COM'} at 127.0.0.1:46364
I20260924 16:15:25.369544 11584 tablet_service.cc:1514] Processing DeleteTablet for tablet d2e72fa7decb4bf89ea1840fbfa543bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:15:25 UTC) from {username='kudu', principal='kudu/127.29.234.62@KRBTEST.COM'} at 127.0.0.1:55688
I20260924 16:15:25.375074 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 11341
I20260924 16:15:25.385835 11840 tablet_replica.cc:330] T d2e72fa7decb4bf89ea1840fbfa543bc P 0190b44255744fdd99cfb02c0951ec26: stopping tablet replica
I20260924 16:15:25.386474 11840 raft_consensus.cc:2229] T d2e72fa7decb4bf89ea1840fbfa543bc P 0190b44255744fdd99cfb02c0951ec26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:15:25.387193 11840 raft_consensus.cc:2258] T d2e72fa7decb4bf89ea1840fbfa543bc P 0190b44255744fdd99cfb02c0951ec26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:15:25.389104 11839 tablet_replica.cc:330] T d2e72fa7decb4bf89ea1840fbfa543bc P c5d39193651b4ebeb5b0ee4a56f32173: stopping tablet replica
I20260924 16:15:25.389609 11840 ts_tablet_manager.cc:1905] T d2e72fa7decb4bf89ea1840fbfa543bc P 0190b44255744fdd99cfb02c0951ec26: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:15:25.389670 11839 raft_consensus.cc:2229] T d2e72fa7decb4bf89ea1840fbfa543bc P c5d39193651b4ebeb5b0ee4a56f32173 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:15:25.390280 11839 raft_consensus.cc:2258] T d2e72fa7decb4bf89ea1840fbfa543bc P c5d39193651b4ebeb5b0ee4a56f32173 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:15:25.392664 11839 ts_tablet_manager.cc:1905] T d2e72fa7decb4bf89ea1840fbfa543bc P c5d39193651b4ebeb5b0ee4a56f32173: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:15:25.396669 11810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:25.376204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46370 (local address 127.29.234.1:35129)
0924 16:15:25.376393 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:25.376400 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:25.376544 (+ 144us) server_negotiation.cc:407] Connection header received
0924 16:15:25.376791 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:25.376807 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:25.376897 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:25.377015 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:25.379280 (+ 2265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:25.380549 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:25.383469 (+ 2920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:25.384379 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:25.384544 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:25.388160 (+ 3616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:25.388179 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:25.388200 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:15:25.388243 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:25.392267 (+ 4024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:25.393205 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:25.393214 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:25.393218 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:25.393397 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:25.393870 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:25.393875 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:25.393878 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:25.394374 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:25.394718 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:25.396001 (+ 1283us) server_negotiation.cc:299] Negotiation successful
0924 16:15:25.396258 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 16:15:25.397240 11840 ts_tablet_manager.cc:1918] T d2e72fa7decb4bf89ea1840fbfa543bc P 0190b44255744fdd99cfb02c0951ec26: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:15:25.397495 11840 log.cc:1196] T d2e72fa7decb4bf89ea1840fbfa543bc P 0190b44255744fdd99cfb02c0951ec26: Deleting WAL directory at /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-1/wal/wals/d2e72fa7decb4bf89ea1840fbfa543bc
I20260924 16:15:25.397791 11499 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:15:25.398106 11840 ts_tablet_manager.cc:1939] T d2e72fa7decb4bf89ea1840fbfa543bc P 0190b44255744fdd99cfb02c0951ec26: Deleting consensus metadata
I20260924 16:15:25.399056 11838 tablet_replica.cc:330] T d2e72fa7decb4bf89ea1840fbfa543bc P 6712eda39e82456db9a5d7e53c096263: stopping tablet replica
I20260924 16:15:25.399539 11838 raft_consensus.cc:2229] T d2e72fa7decb4bf89ea1840fbfa543bc P 6712eda39e82456db9a5d7e53c096263 [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:15:25.400254 11838 raft_consensus.cc:2258] T d2e72fa7decb4bf89ea1840fbfa543bc P 6712eda39e82456db9a5d7e53c096263 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:15:25.400552 11289 catalog_manager.cc:4841] TS 0190b44255744fdd99cfb02c0951ec26 (127.29.234.2:36011): tablet d2e72fa7decb4bf89ea1840fbfa543bc (table test-table [id=47a3b02fc56842dc8faa6e20e333ae90]) successfully deleted
I20260924 16:15:25.402596 11838 ts_tablet_manager.cc:1905] T d2e72fa7decb4bf89ea1840fbfa543bc P 6712eda39e82456db9a5d7e53c096263: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:15:25.409929 11838 ts_tablet_manager.cc:1918] T d2e72fa7decb4bf89ea1840fbfa543bc P 6712eda39e82456db9a5d7e53c096263: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:15:25.410107 11838 log.cc:1196] T d2e72fa7decb4bf89ea1840fbfa543bc P 6712eda39e82456db9a5d7e53c096263: Deleting WAL directory at /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-2/wal/wals/d2e72fa7decb4bf89ea1840fbfa543bc
I20260924 16:15:25.410669 11838 ts_tablet_manager.cc:1939] T d2e72fa7decb4bf89ea1840fbfa543bc P 6712eda39e82456db9a5d7e53c096263: Deleting consensus metadata
I20260924 16:15:25.412533 11289 catalog_manager.cc:4841] TS 6712eda39e82456db9a5d7e53c096263 (127.29.234.3:35745): tablet d2e72fa7decb4bf89ea1840fbfa543bc (table test-table [id=47a3b02fc56842dc8faa6e20e333ae90]) successfully deleted
I20260924 16:15:25.494195 11839 ts_tablet_manager.cc:1918] T d2e72fa7decb4bf89ea1840fbfa543bc P c5d39193651b4ebeb5b0ee4a56f32173: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:15:25.494388 11839 log.cc:1196] T d2e72fa7decb4bf89ea1840fbfa543bc P c5d39193651b4ebeb5b0ee4a56f32173: Deleting WAL directory at /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790266323169505-30632-0/minicluster-data/ts-0/wal/wals/d2e72fa7decb4bf89ea1840fbfa543bc
I20260924 16:15:25.494927 11839 ts_tablet_manager.cc:1939] T d2e72fa7decb4bf89ea1840fbfa543bc P c5d39193651b4ebeb5b0ee4a56f32173: Deleting consensus metadata
I20260924 16:15:25.496927 11289 catalog_manager.cc:4841] TS c5d39193651b4ebeb5b0ee4a56f32173 (127.29.234.1:35129): tablet d2e72fa7decb4bf89ea1840fbfa543bc (table test-table [id=47a3b02fc56842dc8faa6e20e333ae90]) successfully deleted
I20260924 16:15:25.497618 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 11341
I20260924 16:15:25.522325 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 11508
I20260924 16:15:25.537930 11813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:25.523164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55700 (local address 127.29.234.2:36011)
0924 16:15:25.523371 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:25.523375 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:25.523393 (+ 18us) server_negotiation.cc:407] Connection header received
0924 16:15:25.523556 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:25.523560 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:25.523617 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:25.523704 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:25.525455 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:25.526196 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:25.528066 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:25.528517 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:25.528607 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:25.532094 (+ 3487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:25.532107 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:25.532125 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:15:25.532155 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:25.535106 (+ 2951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:25.535949 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:25.535957 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:25.535959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:25.536090 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:25.536478 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:25.536481 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:25.536483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:25.536836 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:25.537062 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:25.537478 (+ 416us) server_negotiation.cc:299] Negotiation successful
0924 16:15:25.537651 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 16:15:25.538877 11645 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:15:25.628621 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 11508
I20260924 16:15:25.652288 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 11654
I20260924 16:15:25.668970 11812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:25.653322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37104 (local address 127.29.234.3:35745)
0924 16:15:25.653625 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:25.653631 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:25.653658 (+ 27us) server_negotiation.cc:407] Connection header received
0924 16:15:25.653803 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:25.653811 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:25.653914 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:25.654030 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:25.655402 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:25.656677 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:25.658114 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:25.658829 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:25.659005 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:25.661739 (+ 2734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:25.661756 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:25.661779 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:15:25.661824 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:25.665929 (+ 4105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:25.666645 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:25.666656 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:25.666659 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:25.666842 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:25.667232 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:25.667239 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:25.667245 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:25.667747 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:25.668070 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:25.668318 (+ 248us) server_negotiation.cc:299] Negotiation successful
0924 16:15:25.668547 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182}
I20260924 16:15:25.669843 11782 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:15:25.794945 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 11654
I20260924 16:15:25.815382 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 11270
I20260924 16:15:25.816645 11334 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:15:25.938086 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 11270
I20260924 16:15:25.964051 30632 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 16:15:26.299371 30632 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 16:15:26.299567 30632 mini_ranger.cc:66] Stopping Ranger...
I20260924 16:15:26.645768 30632 mini_ranger.cc:68] Stopped Ranger
2026-09-24 16:15:26.646 UTC [10304] LOG: received smart shutdown request
2026-09-24 16:15:26.649 UTC [10304] LOG: background worker "logical replication launcher" (PID 10313) exited with exit code 1
2026-09-24 16:15:26.649 UTC [10308] LOG: shutting down
2026-09-24 16:15:26.823 UTC [10304] LOG: database system is shut down
2026-09-24T16:15:26Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (53871 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266323169505-30632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:15:27 dist-test-slave-v5j9 krb5kdc[11854](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:15:27 dist-test-slave-v5j9 krb5kdc[11854](info): set up 2 sockets
Sep 24 16:15:27 dist-test-slave-v5j9 krb5kdc[11854](info): commencing operation
krb5kdc: starting...
W20260924 16:15:28.073300 30632 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.001s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Sep 24 16:15:28 dist-test-slave-v5j9 krb5kdc[11854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:15:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:15:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.234.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab.
Entry for principal kudu/127.29.234.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab.
I20260924 16:15:28.312979 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266323169505-30632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266323169505-30632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266323169505-30632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:39487
--webserver_interface=127.29.234.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab
--principal=kudu/127.29.234.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:45083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.234.62:39487,127.29.234.61:42435,127.29.234.60:38803
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:15:28.645844 11867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:15:28.646154 11867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:15:28.646248 11867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:15:28.653430 11867 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:15:28.653606 11867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:15:28.653707 11867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:15:28.653767 11867 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:15:28.653818 11867 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:15:28.663775 11867 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:45083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266323169505-30632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266323169505-30632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.234.62:39487,127.29.234.61:42435,127.29.234.60:38803
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab
--principal=kudu/127.29.234.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:39487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.234.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266323169505-30632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:15:28.665616 11867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:15:28.667003 11867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:15:28.678936 11873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:28.679054 11875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:28.679803 11872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:15:28.680342 11867 server_base.cc:1034] running on GCE node
I20260924 16:15:28.681160 11867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:15:28.682686 11867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:15:28.683964 11867 hybrid_clock.cc:648] HybridClock initialized: now 1790266528683903 us; error 38 us; skew 500 ppm
I20260924 16:15:28.684441 11867 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:15:28 dist-test-slave-v5j9 krb5kdc[11854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266528, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:15:28.693032 11867 init.cc:377] Logged in from keytab as kudu/127.29.234.62@KRBTEST.COM (short username kudu)
I20260924 16:15:28.694582 11867 webserver.cc:460] Webserver started at http://127.29.234.62:32873/ using document root <none> and password file <none>
I20260924 16:15:28.695302 11867 fs_manager.cc:359] Metadata directory not provided
I20260924 16:15:28.695444 11867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:15:28.695762 11867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:15:28.698837 11867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266323169505-30632-0/minicluster-data/master-0/data/instance:
uuid: "3390d4a9d3be457bb6854f3b63402763"
format_stamp: "Formatted at 2026-09-24 16:15:28 on dist-test-slave-v5j9"
server_key: "f573a24df8858e415f214fe0e5d59bdf"
server_key_iv: "17ba04a4a17daf670c61c9175a2249e1"
server_key_version: "encryptionkey@0"
I20260924 16:15:28.699653 11867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266323169505-30632-0/minicluster-data/master-0/wal/instance:
uuid: "3390d4a9d3be457bb6854f3b63402763"
format_stamp: "Formatted at 2026-09-24 16:15:28 on dist-test-slave-v5j9"
server_key: "f573a24df8858e415f214fe0e5d59bdf"
server_key_iv: "17ba04a4a17daf670c61c9175a2249e1"
server_key_version: "encryptionkey@0"
I20260924 16:15:28.704823 11867 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 16:15:28.708195 11882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:28.709507 11867 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:15:28.709683 11867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266323169505-30632-0/minicluster-data/master-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266323169505-30632-0/minicluster-data/master-0/wal
uuid: "3390d4a9d3be457bb6854f3b63402763"
format_stamp: "Formatted at 2026-09-24 16:15:28 on dist-test-slave-v5j9"
server_key: "f573a24df8858e415f214fe0e5d59bdf"
server_key_iv: "17ba04a4a17daf670c61c9175a2249e1"
server_key_version: "encryptionkey@0"
I20260924 16:15:28.709887 11867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266323169505-30632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266323169505-30632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266323169505-30632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:15:28.737177 11867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:15:28.741679 11867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:15:28.741988 11867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:15:28.760949 11867 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.62:39487
I20260924 16:15:28.760985 11934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.62:39487 every 8 connection(s)
I20260924 16:15:28.762411 11867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
I20260924 16:15:28.765772 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 11867
I20260924 16:15:28.766023 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266323169505-30632-0/minicluster-data/master-0/wal/instance
I20260924 16:15:28.766315 11935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:15:28.766777 30632 external_mini_cluster.cc:1428] Setting key df598867d2afa46b750b65cacfffb1f5
I20260924 16:15:28.772960 11935 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.234.62" port: 39487 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.29.234.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.61@KRBTEST.COM" created.
Sep 24 16:15:28 dist-test-slave-v5j9 krb5kdc[11854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266528, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.62@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:15:28.800161 11938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:28.777011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:39487 (local address 127.0.0.1:41090)
0924 16:15:28.777868 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:28.777924 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:28.780021 (+ 2097us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:28.781107 (+ 1086us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:28.781112 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:28.781775 (+ 663us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:15:28.782137 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:28.782156 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:28.784645 (+ 2489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:28.784649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:28.785820 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:28.785833 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:28.787212 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:28.787218 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:28.787355 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:28.788052 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:15:28.788079 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:15:28.792159 (+ 4080us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:15:28.795608 (+ 3449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:15:28.795621 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:15:28.795634 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:15:28.796039 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:15:28.796467 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:15:28.796473 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:15:28.796477 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:15:28.796641 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:15:28.797534 (+ 893us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:15:28.797543 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:15:28.797766 (+ 223us) client_negotiation.cc:769] Sending connection context
0924 16:15:28.798021 (+ 255us) client_negotiation.cc:240] Negotiation successful
0924 16:15:28.798712 (+ 691us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"mutex_wait_us":200,"thread_start_us":380,"threads_started":1}
I20260924 16:15:28.800203 11939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:28.778361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41090 (local address 127.29.234.62:39487)
0924 16:15:28.779097 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:28.779107 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:28.779163 (+ 56us) server_negotiation.cc:407] Connection header received
0924 16:15:28.780507 (+ 1344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:28.780518 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:28.780857 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:28.781085 (+ 228us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:28.782363 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:28.784478 (+ 2115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:28.785969 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:28.786965 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:28.787172 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:28.792431 (+ 5259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:28.792440 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:28.792455 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:15:28.792491 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:28.795442 (+ 2951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:28.796199 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:28.796206 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:28.796208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:28.796354 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:28.796764 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:28.796773 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:28.796775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:28.797333 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:28.797604 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:28.798169 (+ 565us) server_negotiation.cc:299] Negotiation successful
0924 16:15:28.798593 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":506,"thread_start_us":329,"threads_started":1}
I20260924 16:15:28.803478 11935 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.234.61" port: 42435 } has no permanent_uuid. Determining permanent_uuid...
I20260924 16:15:28.804760 11938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:28.804324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.61:42435 (local address 127.0.0.1:60144)
0924 16:15:28.804514 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:28.804635 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.61:42435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260924 16:15:28.805104 11885 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.234.61:42435: connect: Connection refused (error 111)
I20260924 16:15:28.807000 11938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:28.806476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.61:42435 (local address 127.0.0.1:60160)
0924 16:15:28.806681 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:28.806785 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.61:42435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260924 16:15:28.807216 11935 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.234.61:42435: Network error: Client connection negotiation failed: client connection to 127.29.234.61:42435: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.61.keytab.
Entry for principal kudu/127.29.234.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.61.keytab.
I20260924 16:15:28.820619 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266323169505-30632-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266323169505-30632-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266323169505-30632-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266323169505-30632-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.61:42435
--webserver_interface=127.29.234.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.61.keytab
--principal=kudu/127.29.234.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:45083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.234.62:39487,127.29.234.61:42435,127.29.234.60:38803
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:15:28.856652 11935 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.234.61" port: 42435 } attempt: 1
I20260924 16:15:28.858014 11938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:28.857467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.61:42435 (local address 127.0.0.1:60174)
0924 16:15:28.857689 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:28.857848 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.61:42435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260924 16:15:28.859093 11938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:28.858649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.61:42435 (local address 127.0.0.1:60182)
0924 16:15:28.858815 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:28.858900 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.61:42435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260924 16:15:28.859328 11935 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.234.61:42435: Network error: Client connection negotiation failed: client connection to 127.29.234.61:42435: connect: Connection refused (error 111)
I20260924 16:15:28.927630 11935 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.234.61" port: 42435 } attempt: 2
I20260924 16:15:28.929028 11938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:28.928403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.61:42435 (local address 127.0.0.1:60198)
0924 16:15:28.928670 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:28.928815 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.61:42435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260924 16:15:28.930152 11938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:28.929724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.61:42435 (local address 127.0.0.1:60210)
0924 16:15:28.929904 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:28.930002 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.61:42435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260924 16:15:28.930369 11935 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.234.61:42435: Network error: Client connection negotiation failed: client connection to 127.29.234.61:42435: connect: Connection refused (error 111)
I20260924 16:15:29.021680 11935 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.234.61" port: 42435 } attempt: 3
I20260924 16:15:29.023032 11938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:29.022445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.61:42435 (local address 127.0.0.1:60218)
0924 16:15:29.022692 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:29.022833 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.61:42435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260924 16:15:29.024132 11938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:29.023724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.61:42435 (local address 127.0.0.1:60234)
0924 16:15:29.023891 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:29.023976 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.61:42435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260924 16:15:29.024343 11935 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.234.61:42435: Network error: Client connection negotiation failed: client connection to 127.29.234.61:42435: connect: Connection refused (error 111)
W20260924 16:15:29.149892 11942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:15:29.150226 11942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:15:29.150391 11942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:15:29.157612 11942 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:15:29.157719 11942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:15:29.157840 11942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:15:29.157943 11942 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:15:29.158030 11942 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:15:29.167637 11935 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.234.61" port: 42435 } attempt: 4
I20260924 16:15:29.168012 11942 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:45083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266323169505-30632-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266323169505-30632-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.29.234.62:39487,127.29.234.61:42435,127.29.234.60:38803
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.61.keytab
--principal=kudu/127.29.234.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.61:42435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266323169505-30632-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.29.234.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266323169505-30632-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:15:29.169083 11938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:29.168444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.61:42435 (local address 127.0.0.1:60242)
0924 16:15:29.168815 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:29.168918 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.61:42435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173}
I20260924 16:15:29.169767 11942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:15:29.170298 11938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:29.169855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.61:42435 (local address 127.0.0.1:60244)
0924 16:15:29.170036 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:29.170148 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.61:42435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260924 16:15:29.170528 11935 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.234.61:42435: Network error: Client connection negotiation failed: client connection to 127.29.234.61:42435: connect: Connection refused (error 111)
I20260924 16:15:29.171339 11942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:15:29.183689 11947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:29.183728 11950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:29.183638 11948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:15:29.184723 11942 server_base.cc:1034] running on GCE node
I20260924 16:15:29.185729 11942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:15:29.187331 11942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:15:29.188566 11942 hybrid_clock.cc:648] HybridClock initialized: now 1790266529188512 us; error 51 us; skew 500 ppm
I20260924 16:15:29.188943 11942 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:15:29 dist-test-slave-v5j9 krb5kdc[11854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266529, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:15:29.192291 11942 init.cc:377] Logged in from keytab as kudu/127.29.234.61@KRBTEST.COM (short username kudu)
I20260924 16:15:29.194061 11942 webserver.cc:460] Webserver started at http://127.29.234.61:33229/ using document root <none> and password file <none>
I20260924 16:15:29.194821 11942 fs_manager.cc:359] Metadata directory not provided
I20260924 16:15:29.194975 11942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:15:29.195305 11942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:15:29.198460 11942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266323169505-30632-0/minicluster-data/master-1/data/instance:
uuid: "cbb04676217f461282dc54deaf91a1a8"
format_stamp: "Formatted at 2026-09-24 16:15:29 on dist-test-slave-v5j9"
server_key: "8da499e481d9d303c85b4ac6d9c4ee80"
server_key_iv: "fd5d697f4036e9c5bcca480047b901aa"
server_key_version: "encryptionkey@0"
I20260924 16:15:29.199285 11942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266323169505-30632-0/minicluster-data/master-1/wal/instance:
uuid: "cbb04676217f461282dc54deaf91a1a8"
format_stamp: "Formatted at 2026-09-24 16:15:29 on dist-test-slave-v5j9"
server_key: "8da499e481d9d303c85b4ac6d9c4ee80"
server_key_iv: "fd5d697f4036e9c5bcca480047b901aa"
server_key_version: "encryptionkey@0"
I20260924 16:15:29.204963 11942 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 16:15:29.208665 11957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:29.210202 11942 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:15:29.210407 11942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266323169505-30632-0/minicluster-data/master-1/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266323169505-30632-0/minicluster-data/master-1/wal
uuid: "cbb04676217f461282dc54deaf91a1a8"
format_stamp: "Formatted at 2026-09-24 16:15:29 on dist-test-slave-v5j9"
server_key: "8da499e481d9d303c85b4ac6d9c4ee80"
server_key_iv: "fd5d697f4036e9c5bcca480047b901aa"
server_key_version: "encryptionkey@0"
I20260924 16:15:29.210613 11942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266323169505-30632-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266323169505-30632-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266323169505-30632-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:15:29.231372 11942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:15:29.235423 11942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:15:29.235718 11942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:15:29.254717 11942 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.61:42435
I20260924 16:15:29.254755 12008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.61:42435 every 8 connection(s)
I20260924 16:15:29.256088 11942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266323169505-30632-0/minicluster-data/master-1/data/info.pb
I20260924 16:15:29.259791 12009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:15:29.263087 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 11942
I20260924 16:15:29.263283 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266323169505-30632-0/minicluster-data/master-1/wal/instance
I20260924 16:15:29.263844 30632 external_mini_cluster.cc:1428] Setting key a78eb3ceabf3f929e27160ecf3eec4aa
I20260924 16:15:29.268844 12009 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.234.62" port: 39487 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 16:15:29 dist-test-slave-v5j9 krb5kdc[11854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266529, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.61@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.234.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.60@KRBTEST.COM" created.
I20260924 16:15:29.290927 11939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:29.272441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41092 (local address 127.29.234.62:39487)
0924 16:15:29.272673 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:29.272678 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:29.274235 (+ 1557us) server_negotiation.cc:407] Connection header received
0924 16:15:29.276149 (+ 1914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:29.276155 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:29.276259 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:29.276377 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:29.278020 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:29.278698 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:29.280265 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:29.280686 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:29.280822 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:29.284600 (+ 3778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:29.284606 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:29.284609 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:15:29.284643 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:29.287631 (+ 2988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:29.288482 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:29.288487 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:29.288489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:29.288611 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:29.289098 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:29.289101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:29.289103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:29.289450 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:29.289572 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:29.290525 (+ 953us) server_negotiation.cc:299] Negotiation successful
0924 16:15:29.290698 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 16:15:29.292573 12012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:29.273085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:39487 (local address 127.0.0.1:41092)
0924 16:15:29.274048 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:29.274087 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:29.275857 (+ 1770us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:29.276637 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:29.276646 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:29.277176 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:15:29.277836 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:29.277855 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:29.278868 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:29.278872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:29.280107 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:29.280118 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:29.280935 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:29.280942 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:29.281275 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:29.282025 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:15:29.282048 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:15:29.284338 (+ 2290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:15:29.287777 (+ 3439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:15:29.287787 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:15:29.287804 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:15:29.288340 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:15:29.288733 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:15:29.288739 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:15:29.288743 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:15:29.288981 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:15:29.289578 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:15:29.289588 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:15:29.290108 (+ 520us) client_negotiation.cc:769] Sending connection context
0924 16:15:29.290444 (+ 336us) client_negotiation.cc:240] Negotiation successful
0924 16:15:29.291002 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"thread_start_us":433,"threads_started":1}
I20260924 16:15:29.294564 12009 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.234.61" port: 42435 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 16:15:29 dist-test-slave-v5j9 krb5kdc[11854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266529, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.61@KRBTEST.COM for kudu/127.29.234.61@KRBTEST.COM
I20260924 16:15:29.311201 12012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:29.295395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.61:42435 (local address 127.0.0.1:60250)
0924 16:15:29.295604 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:29.295625 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:29.295736 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:29.296622 (+ 886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:29.296625 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:29.296979 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:15:29.297112 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:29.297120 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:29.299249 (+ 2129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:29.299254 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:29.300121 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:29.300163 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:29.301495 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:29.301500 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:29.301699 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:29.302298 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:15:29.302315 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:15:29.304300 (+ 1985us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:15:29.308753 (+ 4453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:15:29.308757 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:15:29.308759 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:15:29.309105 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:15:29.309587 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:15:29.309590 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:15:29.309593 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:15:29.309738 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:15:29.310514 (+ 776us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:15:29.310517 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:15:29.310625 (+ 108us) client_negotiation.cc:769] Sending connection context
0924 16:15:29.310739 (+ 114us) client_negotiation.cc:240] Negotiation successful
0924 16:15:29.310931 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"mutex_wait_us":24}
I20260924 16:15:29.311720 12013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:29.295497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60250 (local address 127.29.234.61:42435)
0924 16:15:29.296123 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:29.296129 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:29.296154 (+ 25us) server_negotiation.cc:407] Connection header received
0924 16:15:29.296199 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:29.296205 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:29.296454 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:29.296624 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:29.297267 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:29.299072 (+ 1805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:29.300307 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:29.301262 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:29.301494 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:29.304523 (+ 3029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:29.304539 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:29.304561 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:15:29.304601 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:29.308605 (+ 4004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:29.309249 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:29.309262 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:29.309267 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:29.309466 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:29.309854 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:29.309862 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:29.309867 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:29.310380 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:29.310590 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:29.310936 (+ 346us) server_negotiation.cc:299] Negotiation successful
0924 16:15:29.311225 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":466,"thread_start_us":325,"threads_started":1}
I20260924 16:15:29.314457 12009 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.234.60" port: 38803 } has no permanent_uuid. Determining permanent_uuid...
I20260924 16:15:29.315764 12012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:29.315221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.60:38803 (local address 127.0.0.1:53298)
0924 16:15:29.315396 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:29.315558 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.60:38803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 16:15:29.316032 11958 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.234.60:38803: connect: Connection refused (error 111)
I20260924 16:15:29.317659 12012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:29.317242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.60:38803 (local address 127.0.0.1:53308)
0924 16:15:29.317445 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:29.317523 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.60:38803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 16:15:29.317903 12009 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.234.60:38803: Network error: Client connection negotiation failed: client connection to 127.29.234.60:38803: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.60.keytab.
Entry for principal kudu/127.29.234.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.60.keytab.
I20260924 16:15:29.320675 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266323169505-30632-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266323169505-30632-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266323169505-30632-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266323169505-30632-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.60:38803
--webserver_interface=127.29.234.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.60.keytab
--principal=kudu/127.29.234.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:45083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.234.62:39487,127.29.234.61:42435,127.29.234.60:38803
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:15:29.367206 12009 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.234.60" port: 38803 } attempt: 1
I20260924 16:15:29.368466 12012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:29.368002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.60:38803 (local address 127.0.0.1:53312)
0924 16:15:29.368208 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:29.368316 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.60:38803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260924 16:15:29.369568 12012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:29.369146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.60:38803 (local address 127.0.0.1:53328)
0924 16:15:29.369326 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:29.369454 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.60:38803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260924 16:15:29.369808 12009 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.234.60:38803: Network error: Client connection negotiation failed: client connection to 127.29.234.60:38803: connect: Connection refused (error 111)
I20260924 16:15:29.438038 12009 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.234.60" port: 38803 } attempt: 2
I20260924 16:15:29.439374 12012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:29.438834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.60:38803 (local address 127.0.0.1:53330)
0924 16:15:29.439042 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:29.439181 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.60:38803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260924 16:15:29.440402 12012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:29.440038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.60:38803 (local address 127.0.0.1:53334)
0924 16:15:29.440195 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:29.440278 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.60:38803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260924 16:15:29.440610 12009 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.234.60:38803: Network error: Client connection negotiation failed: client connection to 127.29.234.60:38803: connect: Connection refused (error 111)
I20260924 16:15:29.469825 11935 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.234.61" port: 42435 } attempt: 5
Sep 24 16:15:29 dist-test-slave-v5j9 krb5kdc[11854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266528, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.62@KRBTEST.COM for kudu/127.29.234.61@KRBTEST.COM
I20260924 16:15:29.485548 12013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:29.470713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60256 (local address 127.29.234.61:42435)
0924 16:15:29.470899 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:29.470904 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:29.470925 (+ 21us) server_negotiation.cc:407] Connection header received
0924 16:15:29.471161 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:29.471165 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:29.471236 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:29.471320 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:29.472527 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:29.473266 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:29.474974 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:29.475445 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:29.475540 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:29.479604 (+ 4064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:29.479609 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:29.479613 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:15:29.479651 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:29.482388 (+ 2737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:29.483348 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:29.483353 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:29.483355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:29.483539 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:29.484021 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:29.484024 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:29.484027 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:29.484353 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:29.484468 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:29.485017 (+ 549us) server_negotiation.cc:299] Negotiation successful
0924 16:15:29.485235 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 16:15:29.485761 11938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:29.470594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.61:42435 (local address 127.0.0.1:60256)
0924 16:15:29.470838 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:29.470865 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:29.471017 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:29.471408 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:29.471416 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:29.472045 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:15:29.472342 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:29.472361 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:29.473428 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:29.473434 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:29.474809 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:29.474837 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:29.475598 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:29.475605 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:29.475862 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:29.476551 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:15:29.476576 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:15:29.479404 (+ 2828us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:15:29.482542 (+ 3138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:15:29.482550 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:15:29.482556 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:15:29.483184 (+ 628us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:15:29.483676 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:15:29.483684 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:15:29.483688 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:15:29.483899 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:15:29.484489 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:15:29.484495 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:15:29.484707 (+ 212us) client_negotiation.cc:769] Sending connection context
0924 16:15:29.484915 (+ 208us) client_negotiation.cc:240] Negotiation successful
0924 16:15:29.485286 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260924 16:15:29.487119 11935 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.234.60" port: 38803 } has no permanent_uuid. Determining permanent_uuid...
I20260924 16:15:29.488500 11938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:29.488018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.60:38803 (local address 127.0.0.1:53344)
0924 16:15:29.488235 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:29.488332 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.60:38803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260924 16:15:29.489607 11938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:29.489156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.60:38803 (local address 127.0.0.1:53358)
0924 16:15:29.489369 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:29.489462 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.60:38803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260924 16:15:29.489902 11935 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.234.60:38803: Network error: Client connection negotiation failed: client connection to 127.29.234.60:38803: connect: Connection refused (error 111)
I20260924 16:15:29.531831 12009 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.234.60" port: 38803 } attempt: 3
I20260924 16:15:29.533027 12012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:29.532540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.60:38803 (local address 127.0.0.1:53370)
0924 16:15:29.532754 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:29.532863 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.60:38803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260924 16:15:29.534219 12012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:29.533783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.60:38803 (local address 127.0.0.1:53376)
0924 16:15:29.534020 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:29.534098 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.60:38803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
W20260924 16:15:29.534606 12009 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.234.60:38803: Network error: Client connection negotiation failed: client connection to 127.29.234.60:38803: connect: Connection refused (error 111)
I20260924 16:15:29.541189 11935 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.234.60" port: 38803 } attempt: 1
I20260924 16:15:29.542454 11938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:29.541903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.60:38803 (local address 127.0.0.1:53390)
0924 16:15:29.542139 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:29.542267 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.60:38803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260924 16:15:29.543563 11938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:29.543186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.60:38803 (local address 127.0.0.1:53398)
0924 16:15:29.543351 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:29.543437 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.60:38803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260924 16:15:29.543821 11935 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.234.60:38803: Network error: Client connection negotiation failed: client connection to 127.29.234.60:38803: connect: Connection refused (error 111)
I20260924 16:15:29.612059 11935 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.234.60" port: 38803 } attempt: 2
I20260924 16:15:29.613333 11938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:29.612771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.60:38803 (local address 127.0.0.1:53400)
0924 16:15:29.613014 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:29.613128 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.60:38803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260924 16:15:29.614607 11938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:29.614176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.60:38803 (local address 127.0.0.1:53406)
0924 16:15:29.614382 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:29.614468 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.60:38803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260924 16:15:29.614768 11935 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.234.60:38803: Network error: Client connection negotiation failed: client connection to 127.29.234.60:38803: connect: Connection refused (error 111)
W20260924 16:15:29.659636 12016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:15:29.659935 12016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:15:29.660037 12016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:15:29.667320 12016 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:15:29.667423 12016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:15:29.667471 12016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:15:29.667528 12016 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:15:29.667572 12016 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:15:29.677388 12016 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:45083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266323169505-30632-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266323169505-30632-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.29.234.62:39487,127.29.234.61:42435,127.29.234.60:38803
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.60.keytab
--principal=kudu/127.29.234.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.60:38803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266323169505-30632-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.29.234.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266323169505-30632-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:15:29.677863 12009 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.234.60" port: 38803 } attempt: 4
I20260924 16:15:29.679054 12012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:29.678568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.60:38803 (local address 127.0.0.1:53418)
0924 16:15:29.678810 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:29.678908 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.60:38803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260924 16:15:29.679066 12016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:15:29.680080 12012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:29.679775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.60:38803 (local address 127.0.0.1:53426)
0924 16:15:29.679905 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:29.679973 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.60:38803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260924 16:15:29.680403 12009 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.234.60:38803: Network error: Client connection negotiation failed: client connection to 127.29.234.60:38803: connect: Connection refused (error 111)
I20260924 16:15:29.680843 12016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:15:29.693747 12021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:29.693748 12024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:29.694020 12022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:15:29.694721 12016 server_base.cc:1034] running on GCE node
I20260924 16:15:29.695518 12016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:15:29.697002 12016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:15:29.698223 12016 hybrid_clock.cc:648] HybridClock initialized: now 1790266529698171 us; error 35 us; skew 500 ppm
I20260924 16:15:29.698585 12016 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:15:29 dist-test-slave-v5j9 krb5kdc[11854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266529, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:15:29.701819 12016 init.cc:377] Logged in from keytab as kudu/127.29.234.60@KRBTEST.COM (short username kudu)
I20260924 16:15:29.703279 12016 webserver.cc:460] Webserver started at http://127.29.234.60:33757/ using document root <none> and password file <none>
I20260924 16:15:29.704041 12016 fs_manager.cc:359] Metadata directory not provided
I20260924 16:15:29.704149 12016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:15:29.704421 12016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:15:29.707494 12016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266323169505-30632-0/minicluster-data/master-2/data/instance:
uuid: "c60caf4257f94463810f97475fd3499f"
format_stamp: "Formatted at 2026-09-24 16:15:29 on dist-test-slave-v5j9"
server_key: "7d27861bda19d343d2bdfa69cb079390"
server_key_iv: "41634eeb90bc0ebd80fdba2b02c5af4e"
server_key_version: "encryptionkey@0"
I20260924 16:15:29.708384 12016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266323169505-30632-0/minicluster-data/master-2/wal/instance:
uuid: "c60caf4257f94463810f97475fd3499f"
format_stamp: "Formatted at 2026-09-24 16:15:29 on dist-test-slave-v5j9"
server_key: "7d27861bda19d343d2bdfa69cb079390"
server_key_iv: "41634eeb90bc0ebd80fdba2b02c5af4e"
server_key_version: "encryptionkey@0"
I20260924 16:15:29.713932 12016 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260924 16:15:29.717691 12031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:29.719094 12016 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:15:29.719307 12016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266323169505-30632-0/minicluster-data/master-2/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266323169505-30632-0/minicluster-data/master-2/wal
uuid: "c60caf4257f94463810f97475fd3499f"
format_stamp: "Formatted at 2026-09-24 16:15:29 on dist-test-slave-v5j9"
server_key: "7d27861bda19d343d2bdfa69cb079390"
server_key_iv: "41634eeb90bc0ebd80fdba2b02c5af4e"
server_key_version: "encryptionkey@0"
I20260924 16:15:29.719511 12016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266323169505-30632-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266323169505-30632-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266323169505-30632-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:15:29.720968 11935 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.234.60" port: 38803 } attempt: 3
I20260924 16:15:29.722096 11938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:29.721638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.60:38803 (local address 127.0.0.1:53428)
0924 16:15:29.721866 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:29.721952 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.60:38803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260924 16:15:29.723071 11938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:29.722715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.60:38803 (local address 127.0.0.1:53442)
0924 16:15:29.722894 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:29.722974 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.60:38803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260924 16:15:29.723274 11935 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.234.60:38803: Network error: Client connection negotiation failed: client connection to 127.29.234.60:38803: connect: Connection refused (error 111)
I20260924 16:15:29.735086 12016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:15:29.739383 12016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:15:29.739681 12016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:15:29.758594 12016 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.60:38803
I20260924 16:15:29.758630 12083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.60:38803 every 8 connection(s)
I20260924 16:15:29.760008 12016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266323169505-30632-0/minicluster-data/master-2/data/info.pb
I20260924 16:15:29.764000 12084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:15:29.765426 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 12016
I20260924 16:15:29.765625 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790266323169505-30632-0/minicluster-data/master-2/wal/instance
I20260924 16:15:29.766357 30632 external_mini_cluster.cc:1428] Setting key 570dac31f033f969f897d043e12db9ba
I20260924 16:15:29.774425 12084 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.234.62" port: 39487 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 16:15:29 dist-test-slave-v5j9 krb5kdc[11854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.234.60@KRBTEST.COM
Sep 24 16:15:29 dist-test-slave-v5j9 krb5kdc[11854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266529, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.60@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:15:29.790506 12087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:29.768432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53446 (local address 127.29.234.60:38803)
0924 16:15:29.769294 (+ 862us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:29.769311 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:29.769395 (+ 84us) server_negotiation.cc:407] Connection header received
0924 16:15:29.771219 (+ 1824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:29.771250 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:29.771791 (+ 541us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:29.772258 (+ 467us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:29.773691 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:29.775712 (+ 2021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:29.777419 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:29.778239 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:29.778987 (+ 748us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:29.782946 (+ 3959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:29.782955 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:29.782965 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:15:29.783001 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:29.786419 (+ 3418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:29.787147 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:29.787162 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:29.787168 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:29.787311 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:29.787688 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:29.787695 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:29.787699 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:29.788205 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:29.788491 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:29.788971 (+ 480us) server_negotiation.cc:299] Negotiation successful
0924 16:15:29.789442 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":370,"threads_started":1}
I20260924 16:15:29.792243 11939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:29.777253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41094 (local address 127.29.234.62:39487)
0924 16:15:29.777467 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:29.777473 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:29.778349 (+ 876us) server_negotiation.cc:407] Connection header received
0924 16:15:29.778636 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:29.778641 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:29.778704 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:29.778809 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:29.780322 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:29.781002 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:29.782535 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:29.782967 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:29.783100 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:29.786643 (+ 3543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:29.786648 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:29.786652 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:15:29.786684 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:29.789335 (+ 2651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:29.790094 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:29.790099 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:29.790102 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:29.790295 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:29.790772 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:29.790776 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:29.790778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:29.791021 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:29.791198 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:29.791643 (+ 445us) server_negotiation.cc:299] Negotiation successful
0924 16:15:29.791866 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 16:15:29.792273 12088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:29.777374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:39487 (local address 127.0.0.1:41094)
0924 16:15:29.778214 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:29.778257 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:29.778463 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:29.778877 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:29.778884 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:29.779773 (+ 889us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:15:29.780163 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:29.780178 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:29.781162 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:29.781168 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:29.782393 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:29.782405 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:29.783171 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:29.783181 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:29.783364 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:29.784053 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:15:29.784083 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:15:29.786448 (+ 2365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:15:29.789506 (+ 3058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:15:29.789517 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:15:29.789524 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:15:29.789963 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:15:29.790412 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:15:29.790417 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:15:29.790420 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:15:29.790596 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:15:29.791154 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:15:29.791162 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:15:29.791340 (+ 178us) client_negotiation.cc:769] Sending connection context
0924 16:15:29.791507 (+ 167us) client_negotiation.cc:240] Negotiation successful
0924 16:15:29.791854 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":429,"threads_started":1}
I20260924 16:15:29.794268 12084 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.234.61" port: 42435 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 16:15:29 dist-test-slave-v5j9 krb5kdc[11854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266529, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.60@KRBTEST.COM for kudu/127.29.234.61@KRBTEST.COM
Sep 24 16:15:29 dist-test-slave-v5j9 krb5kdc[11854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:15:29.808058 12088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:29.795092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.61:42435 (local address 127.0.0.1:60262)
0924 16:15:29.795240 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:29.795259 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:29.795359 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:29.795696 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:29.795701 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:29.796049 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:15:29.796183 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:29.796193 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:29.797110 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:29.797118 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:29.798041 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:29.798053 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:29.799300 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:29.799306 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:29.799502 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:29.800023 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:15:29.800039 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:15:29.802111 (+ 2072us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:15:29.805705 (+ 3594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:15:29.805711 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:15:29.805715 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:15:29.806082 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:15:29.806478 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:15:29.806482 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:15:29.806486 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:15:29.806636 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:15:29.807126 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:15:29.807131 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:15:29.807284 (+ 153us) client_negotiation.cc:769] Sending connection context
0924 16:15:29.807432 (+ 148us) client_negotiation.cc:240] Negotiation successful
0924 16:15:29.807703 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260924 16:15:29.808096 12013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:29.795026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60262 (local address 127.29.234.61:42435)
0924 16:15:29.795236 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:29.795242 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:29.795305 (+ 63us) server_negotiation.cc:407] Connection header received
0924 16:15:29.795488 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:29.795493 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:29.795568 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:29.795667 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:29.796324 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:29.796979 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:29.798178 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:29.798919 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:29.799058 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:29.802311 (+ 3253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:29.802317 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:29.802323 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:15:29.802373 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:29.805582 (+ 3209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:29.806221 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:29.806226 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:29.806230 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:29.806368 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:29.806743 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:29.806750 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:29.806753 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:29.807009 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:29.807161 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:29.807520 (+ 359us) server_negotiation.cc:299] Negotiation successful
0924 16:15:29.807732 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 16:15:29.808583 11939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:29.793802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41098 (local address 127.29.234.62:39487)
0924 16:15:29.793969 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:29.793977 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:29.793996 (+ 19us) server_negotiation.cc:407] Connection header received
0924 16:15:29.794047 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:29.794052 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:29.794118 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:29.794226 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:29.795713 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:29.796446 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:29.797722 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:29.798124 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:29.798268 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:29.803153 (+ 4885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:29.803159 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:29.803163 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:15:29.803198 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:29.805816 (+ 2618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:29.806478 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:29.806484 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:29.806487 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:29.806618 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:29.807012 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:29.807017 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:29.807023 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:29.807288 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:29.807575 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:29.808017 (+ 442us) server_negotiation.cc:299] Negotiation successful
0924 16:15:29.808290 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 16:15:29.809727 12084 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.234.60" port: 38803 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 16:15:29 dist-test-slave-v5j9 krb5kdc[11854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266529, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.60@KRBTEST.COM for kudu/127.29.234.60@KRBTEST.COM
Sep 24 16:15:29 dist-test-slave-v5j9 krb5kdc[11854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.234.61@KRBTEST.COM
I20260924 16:15:29.822752 12087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:29.810885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53454 (local address 127.29.234.60:38803)
0924 16:15:29.811026 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:29.811030 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:29.811042 (+ 12us) server_negotiation.cc:407] Connection header received
0924 16:15:29.811075 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:29.811078 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:29.811156 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:29.811260 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:29.811914 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:29.812645 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:29.813902 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:29.814375 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:29.814510 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:29.817504 (+ 2994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:29.817508 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:29.817511 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:15:29.817537 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:29.820312 (+ 2775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:29.820901 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:29.820907 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:29.820911 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:29.821020 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:29.821382 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:29.821387 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:29.821391 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:29.821666 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:29.821815 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:29.822155 (+ 340us) server_negotiation.cc:299] Negotiation successful
0924 16:15:29.822373 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 16:15:29.822877 12088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:29.810591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.60:38803 (local address 127.0.0.1:53454)
0924 16:15:29.810749 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:29.810764 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:29.810840 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:29.811275 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:29.811280 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:29.811656 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:15:29.811776 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:29.811783 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:29.812769 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:29.812777 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:29.813766 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:29.813778 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:29.814546 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:29.814553 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:29.814749 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:29.815297 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:15:29.815314 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:15:29.817265 (+ 1951us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:15:29.820435 (+ 3170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:15:29.820442 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:15:29.820446 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:15:29.820788 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:15:29.821122 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:15:29.821136 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:15:29.821141 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:15:29.821268 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:15:29.821777 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:15:29.821781 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:15:29.821929 (+ 148us) client_negotiation.cc:769] Sending connection context
0924 16:15:29.822081 (+ 152us) client_negotiation.cc:240] Negotiation successful
0924 16:15:29.822526 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66,"mutex_wait_us":142}
I20260924 16:15:29.827418 12013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:29.810486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60278 (local address 127.29.234.61:42435)
0924 16:15:29.811056 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:29.811060 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:29.811076 (+ 16us) server_negotiation.cc:407] Connection header received
0924 16:15:29.811116 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:29.811120 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:29.811183 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:29.811298 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:29.812661 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:29.813734 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:29.815459 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:29.816033 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:29.816182 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:29.821190 (+ 5008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:29.821195 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:29.821199 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:15:29.821231 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:29.824532 (+ 3301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:29.825256 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:29.825263 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:29.825267 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:29.825480 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:29.825889 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:29.825895 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:29.825899 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:29.826199 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:29.826489 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:29.826821 (+ 332us) server_negotiation.cc:299] Negotiation successful
0924 16:15:29.827076 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473}
I20260924 16:15:29.828291 12084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c60caf4257f94463810f97475fd3499f: Bootstrap starting.
I20260924 16:15:29.832261 12084 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c60caf4257f94463810f97475fd3499f: Neither blocks nor log segments found. Creating new log.
I20260924 16:15:29.832736 30632 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:15:29.833595 12084 log.cc:824] T 00000000000000000000000000000000 P c60caf4257f94463810f97475fd3499f: Log is configured to *not* fsync() on all Append() calls
I20260924 16:15:29.837071 12084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c60caf4257f94463810f97475fd3499f: No bootstrap required, opened a new log
I20260924 16:15:29.851235 12084 raft_consensus.cc:348] T 00000000000000000000000000000000 P c60caf4257f94463810f97475fd3499f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3390d4a9d3be457bb6854f3b63402763" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 39487 } } peers { permanent_uuid: "cbb04676217f461282dc54deaf91a1a8" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 42435 } } peers { permanent_uuid: "c60caf4257f94463810f97475fd3499f" member_type: VOTER last_known_addr { host: "127.29.234.60" port: 38803 } }
I20260924 16:15:29.852078 12084 raft_consensus.cc:374] T 00000000000000000000000000000000 P c60caf4257f94463810f97475fd3499f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:15:29.852298 12084 raft_consensus.cc:729] T 00000000000000000000000000000000 P c60caf4257f94463810f97475fd3499f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c60caf4257f94463810f97475fd3499f, State: Initialized, Role: FOLLOWER
I20260924 16:15:29.853289 12084 consensus_queue.cc:260] T 00000000000000000000000000000000 P c60caf4257f94463810f97475fd3499f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3390d4a9d3be457bb6854f3b63402763" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 39487 } } peers { permanent_uuid: "cbb04676217f461282dc54deaf91a1a8" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 42435 } } peers { permanent_uuid: "c60caf4257f94463810f97475fd3499f" member_type: VOTER last_known_addr { host: "127.29.234.60" port: 38803 } }
I20260924 16:15:29.863973 12097 sys_catalog.cc:455] T 00000000000000000000000000000000 P c60caf4257f94463810f97475fd3499f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3390d4a9d3be457bb6854f3b63402763" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 39487 } } peers { permanent_uuid: "cbb04676217f461282dc54deaf91a1a8" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 42435 } } peers { permanent_uuid: "c60caf4257f94463810f97475fd3499f" member_type: VOTER last_known_addr { host: "127.29.234.60" port: 38803 } } }
I20260924 16:15:29.866362 12013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:29.839252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60294 (local address 127.29.234.61:42435)
0924 16:15:29.839438 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:29.839445 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:29.842066 (+ 2621us) server_negotiation.cc:407] Connection header received
0924 16:15:29.843354 (+ 1288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:29.843362 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:29.843454 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:29.843529 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:29.848618 (+ 5089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:29.850133 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:29.853077 (+ 2944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:29.854034 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:29.854165 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:29.858808 (+ 4643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:29.858815 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:29.858820 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:15:29.858861 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:29.862503 (+ 3642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:29.863458 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:29.863466 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:29.863472 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:29.863643 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:29.864223 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:29.864228 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:29.864231 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:29.864648 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:29.864888 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:29.865649 (+ 761us) server_negotiation.cc:299] Negotiation successful
0924 16:15:29.865922 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 16:15:29.867065 12097 sys_catalog.cc:458] T 00000000000000000000000000000000 P c60caf4257f94463810f97475fd3499f [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:15:29.865646 12084 sys_catalog.cc:564] T 00000000000000000000000000000000 P c60caf4257f94463810f97475fd3499f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:15:29.874419 11939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:29.839298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41112 (local address 127.29.234.62:39487)
0924 16:15:29.839507 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:29.839515 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:29.852698 (+ 13183us) server_negotiation.cc:407] Connection header received
0924 16:15:29.853243 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:29.853250 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:29.853366 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:29.853448 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:29.856642 (+ 3194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:29.858092 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:29.860936 (+ 2844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:29.861910 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:29.862035 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:29.866236 (+ 4201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:29.866243 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:29.866246 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:15:29.866280 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:29.870783 (+ 4503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:29.871666 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:29.871674 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:29.871678 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:29.871857 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:29.872308 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:29.872315 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:29.872319 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:29.872707 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:29.872950 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:29.873569 (+ 619us) server_negotiation.cc:299] Negotiation successful
0924 16:15:29.873903 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 16:15:29.876732 12087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:29.842382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53464 (local address 127.29.234.60:38803)
0924 16:15:29.845080 (+ 2698us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:29.845088 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:29.855372 (+ 10284us) server_negotiation.cc:407] Connection header received
0924 16:15:29.855938 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:29.855945 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:29.856039 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:29.856116 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:29.858193 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:29.859298 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:29.861534 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:29.862315 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:29.862418 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:29.867080 (+ 4662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:29.867089 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:29.867096 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:15:29.867185 (+ 89us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:29.872745 (+ 5560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:29.873619 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:29.873625 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:29.873628 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:29.873801 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:29.874219 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:29.874226 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:29.874231 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:29.874674 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:29.875978 (+ 1304us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:29.876121 (+ 143us) server_negotiation.cc:299] Negotiation successful
0924 16:15:29.876345 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2094}
W20260924 16:15:29.884423 12108 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c60caf4257f94463810f97475fd3499f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 16:15:29.884658 12108 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 16:15:29.876767 12016 master_runner.cc:186] Error getting master registration for 127.29.234.62:39487: OK, instance_id { permanent_uuid: "3390d4a9d3be457bb6854f3b63402763" instance_seqno: 1790266528755728 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 16:15:29.886745 12016 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 16:15:29.900502 11935 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.234.60" port: 38803 } attempt: 4
Sep 24 16:15:29 dist-test-slave-v5j9 krb5kdc[11854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266528, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.62@KRBTEST.COM for kudu/127.29.234.60@KRBTEST.COM
I20260924 16:15:29.916680 11938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:29.901136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.60:38803 (local address 127.0.0.1:53468)
0924 16:15:29.901388 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:29.901419 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:29.901522 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:29.901935 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:29.901939 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:29.902331 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:15:29.902491 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:29.902500 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:29.904024 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:29.904033 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:29.905588 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:29.905605 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:29.906395 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:29.906402 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:29.906593 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:29.907229 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:15:29.907257 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:15:29.909896 (+ 2639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:15:29.913933 (+ 4037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:15:29.913940 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:15:29.913944 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:15:29.914499 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:15:29.915006 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:15:29.915012 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:15:29.915016 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:15:29.915191 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:15:29.915879 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:15:29.915882 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:15:29.916039 (+ 157us) client_negotiation.cc:769] Sending connection context
0924 16:15:29.916152 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 16:15:29.916407 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260924 16:15:29.916945 12087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:29.901286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53468 (local address 127.29.234.60:38803)
0924 16:15:29.901513 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:29.901520 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:29.901540 (+ 20us) server_negotiation.cc:407] Connection header received
0924 16:15:29.901641 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:29.901648 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:29.901727 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:29.901830 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:29.902644 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:29.903840 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:29.905745 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:29.906230 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:29.906324 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:29.910087 (+ 3763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:29.910094 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:29.910099 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:15:29.910144 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:29.913753 (+ 3609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:29.914644 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:29.914657 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:29.914661 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:29.914879 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:29.915349 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:29.915355 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:29.915359 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:29.915745 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:29.915908 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:29.916313 (+ 405us) server_negotiation.cc:299] Negotiation successful
0924 16:15:29.916565 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260924 16:15:29.923158 11935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3390d4a9d3be457bb6854f3b63402763: Bootstrap starting.
I20260924 16:15:29.927639 11935 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3390d4a9d3be457bb6854f3b63402763: Neither blocks nor log segments found. Creating new log.
I20260924 16:15:29.929126 11935 log.cc:824] T 00000000000000000000000000000000 P 3390d4a9d3be457bb6854f3b63402763: Log is configured to *not* fsync() on all Append() calls
I20260924 16:15:29.932714 11935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3390d4a9d3be457bb6854f3b63402763: No bootstrap required, opened a new log
I20260924 16:15:29.944272 11935 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3390d4a9d3be457bb6854f3b63402763 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3390d4a9d3be457bb6854f3b63402763" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 39487 } } peers { permanent_uuid: "cbb04676217f461282dc54deaf91a1a8" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 42435 } } peers { permanent_uuid: "c60caf4257f94463810f97475fd3499f" member_type: VOTER last_known_addr { host: "127.29.234.60" port: 38803 } }
I20260924 16:15:29.944754 11935 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3390d4a9d3be457bb6854f3b63402763 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:15:29.944893 11935 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3390d4a9d3be457bb6854f3b63402763 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3390d4a9d3be457bb6854f3b63402763, State: Initialized, Role: FOLLOWER
I20260924 16:15:29.945665 11935 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3390d4a9d3be457bb6854f3b63402763 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3390d4a9d3be457bb6854f3b63402763" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 39487 } } peers { permanent_uuid: "cbb04676217f461282dc54deaf91a1a8" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 42435 } } peers { permanent_uuid: "c60caf4257f94463810f97475fd3499f" member_type: VOTER last_known_addr { host: "127.29.234.60" port: 38803 } }
I20260924 16:15:29.946674 12111 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3390d4a9d3be457bb6854f3b63402763 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3390d4a9d3be457bb6854f3b63402763" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 39487 } } peers { permanent_uuid: "cbb04676217f461282dc54deaf91a1a8" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 42435 } } peers { permanent_uuid: "c60caf4257f94463810f97475fd3499f" member_type: VOTER last_known_addr { host: "127.29.234.60" port: 38803 } } }
I20260924 16:15:29.947041 12111 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3390d4a9d3be457bb6854f3b63402763 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:15:29.948196 11935 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3390d4a9d3be457bb6854f3b63402763 [sys.catalog]: configured and running, proceeding with master startup.
W20260924 16:15:29.960886 12122 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3390d4a9d3be457bb6854f3b63402763: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 16:15:29.961131 12122 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 16:15:29.959049 11867 master_runner.cc:186] Error getting master registration for 127.29.234.61:42435: OK, instance_id { permanent_uuid: "cbb04676217f461282dc54deaf91a1a8" instance_seqno: 1790266529249290 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 16:15:29.970893 11867 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 16:15:29.979645 12009 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.234.60" port: 38803 } attempt: 5
Sep 24 16:15:29 dist-test-slave-v5j9 krb5kdc[11854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266529, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.61@KRBTEST.COM for kudu/127.29.234.60@KRBTEST.COM
W20260924 16:15:29.991885 12037 tablet.cc:2369] T 00000000000000000000000000000000 P c60caf4257f94463810f97475fd3499f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 16:15:29.994845 11888 tablet.cc:2369] T 00000000000000000000000000000000 P 3390d4a9d3be457bb6854f3b63402763: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:15:29.998376 12012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:29.980400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.60:38803 (local address 127.0.0.1:53478)
0924 16:15:29.980701 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:29.980723 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:29.980858 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:29.981290 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:29.981296 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:29.981861 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:15:29.982083 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:29.982096 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:29.983720 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:29.983726 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:29.985164 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:29.985179 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:29.986504 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:29.986511 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:29.986726 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:29.987684 (+ 958us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:15:29.987708 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:15:29.990591 (+ 2883us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:15:29.995254 (+ 4663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:15:29.995260 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:15:29.995267 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:15:29.995799 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:15:29.996332 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:15:29.996339 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:15:29.996342 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:15:29.996537 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:15:29.997237 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:15:29.997242 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:15:29.997442 (+ 200us) client_negotiation.cc:769] Sending connection context
0924 16:15:29.997620 (+ 178us) client_negotiation.cc:240] Negotiation successful
0924 16:15:29.997957 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171}
I20260924 16:15:29.998404 12087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:29.980513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53478 (local address 127.29.234.60:38803)
0924 16:15:29.980711 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:29.980717 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:29.980808 (+ 91us) server_negotiation.cc:407] Connection header received
0924 16:15:29.980995 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:29.981001 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:29.981096 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:29.981218 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:29.982294 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:29.983519 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:29.985380 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:29.986141 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:29.986271 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:29.990822 (+ 4551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:29.990828 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:29.990834 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:15:29.990881 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:29.995071 (+ 4190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:29.995937 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:29.995943 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:29.995947 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:29.996102 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:29.996670 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:29.996677 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:29.996680 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:29.997082 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:29.997262 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:29.997735 (+ 473us) server_negotiation.cc:299] Negotiation successful
0924 16:15:29.998008 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 16:15:30.005394 12009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbb04676217f461282dc54deaf91a1a8: Bootstrap starting.
I20260924 16:15:30.011366 12009 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cbb04676217f461282dc54deaf91a1a8: Neither blocks nor log segments found. Creating new log.
I20260924 16:15:30.012851 12009 log.cc:824] T 00000000000000000000000000000000 P cbb04676217f461282dc54deaf91a1a8: Log is configured to *not* fsync() on all Append() calls
I20260924 16:15:30.016629 12009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbb04676217f461282dc54deaf91a1a8: No bootstrap required, opened a new log
I20260924 16:15:30.028339 12009 raft_consensus.cc:348] T 00000000000000000000000000000000 P cbb04676217f461282dc54deaf91a1a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3390d4a9d3be457bb6854f3b63402763" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 39487 } } peers { permanent_uuid: "cbb04676217f461282dc54deaf91a1a8" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 42435 } } peers { permanent_uuid: "c60caf4257f94463810f97475fd3499f" member_type: VOTER last_known_addr { host: "127.29.234.60" port: 38803 } }
I20260924 16:15:30.028812 12009 raft_consensus.cc:374] T 00000000000000000000000000000000 P cbb04676217f461282dc54deaf91a1a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:15:30.028934 12009 raft_consensus.cc:729] T 00000000000000000000000000000000 P cbb04676217f461282dc54deaf91a1a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbb04676217f461282dc54deaf91a1a8, State: Initialized, Role: FOLLOWER
I20260924 16:15:30.029707 12009 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbb04676217f461282dc54deaf91a1a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3390d4a9d3be457bb6854f3b63402763" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 39487 } } peers { permanent_uuid: "cbb04676217f461282dc54deaf91a1a8" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 42435 } } peers { permanent_uuid: "c60caf4257f94463810f97475fd3499f" member_type: VOTER last_known_addr { host: "127.29.234.60" port: 38803 } }
I20260924 16:15:30.030738 12125 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbb04676217f461282dc54deaf91a1a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3390d4a9d3be457bb6854f3b63402763" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 39487 } } peers { permanent_uuid: "cbb04676217f461282dc54deaf91a1a8" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 42435 } } peers { permanent_uuid: "c60caf4257f94463810f97475fd3499f" member_type: VOTER last_known_addr { host: "127.29.234.60" port: 38803 } } }
I20260924 16:15:30.031229 12125 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbb04676217f461282dc54deaf91a1a8 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:15:30.032182 12009 sys_catalog.cc:564] T 00000000000000000000000000000000 P cbb04676217f461282dc54deaf91a1a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:15:30.051450 11942 master_runner.cc:263] No leader master found from master cbb04676217f461282dc54deaf91a1a8
I20260924 16:15:30.051625 11942 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260924 16:15:30.054538 12136 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cbb04676217f461282dc54deaf91a1a8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 16:15:30.054765 12136 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260924 16:15:30.238502 11963 tablet.cc:2369] T 00000000000000000000000000000000 P cbb04676217f461282dc54deaf91a1a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:15:30.271040 12097 raft_consensus.cc:482] T 00000000000000000000000000000000 P c60caf4257f94463810f97475fd3499f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:15:30.271456 12097 raft_consensus.cc:504] T 00000000000000000000000000000000 P c60caf4257f94463810f97475fd3499f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3390d4a9d3be457bb6854f3b63402763" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 39487 } } peers { permanent_uuid: "cbb04676217f461282dc54deaf91a1a8" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 42435 } } peers { permanent_uuid: "c60caf4257f94463810f97475fd3499f" member_type: VOTER last_known_addr { host: "127.29.234.60" port: 38803 } }
I20260924 16:15:30.273291 12097 leader_election.cc:288] T 00000000000000000000000000000000 P c60caf4257f94463810f97475fd3499f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3390d4a9d3be457bb6854f3b63402763 (127.29.234.62:39487), cbb04676217f461282dc54deaf91a1a8 (127.29.234.61:42435)
I20260924 16:15:30.275178 11984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c60caf4257f94463810f97475fd3499f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbb04676217f461282dc54deaf91a1a8" is_pre_election: true
I20260924 16:15:30.275919 11984 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cbb04676217f461282dc54deaf91a1a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c60caf4257f94463810f97475fd3499f in term 0.
I20260924 16:15:30.277177 12034 leader_election.cc:302] T 00000000000000000000000000000000 P c60caf4257f94463810f97475fd3499f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c60caf4257f94463810f97475fd3499f, cbb04676217f461282dc54deaf91a1a8; no voters:
I20260924 16:15:30.277825 12097 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c60caf4257f94463810f97475fd3499f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:15:30.278005 12097 raft_consensus.cc:482] T 00000000000000000000000000000000 P c60caf4257f94463810f97475fd3499f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:15:30.278153 12097 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c60caf4257f94463810f97475fd3499f [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:15:30.281867 11909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c60caf4257f94463810f97475fd3499f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3390d4a9d3be457bb6854f3b63402763" is_pre_election: true
I20260924 16:15:30.282487 11909 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3390d4a9d3be457bb6854f3b63402763 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c60caf4257f94463810f97475fd3499f in term 0.
I20260924 16:15:30.285040 12097 raft_consensus.cc:504] T 00000000000000000000000000000000 P c60caf4257f94463810f97475fd3499f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3390d4a9d3be457bb6854f3b63402763" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 39487 } } peers { permanent_uuid: "cbb04676217f461282dc54deaf91a1a8" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 42435 } } peers { permanent_uuid: "c60caf4257f94463810f97475fd3499f" member_type: VOTER last_known_addr { host: "127.29.234.60" port: 38803 } }
I20260924 16:15:30.286402 11909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c60caf4257f94463810f97475fd3499f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3390d4a9d3be457bb6854f3b63402763"
I20260924 16:15:30.286615 11909 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3390d4a9d3be457bb6854f3b63402763 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:15:30.290661 11909 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3390d4a9d3be457bb6854f3b63402763 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c60caf4257f94463810f97475fd3499f in term 1.
I20260924 16:15:30.291182 12033 leader_election.cc:302] T 00000000000000000000000000000000 P c60caf4257f94463810f97475fd3499f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3390d4a9d3be457bb6854f3b63402763, c60caf4257f94463810f97475fd3499f; no voters:
I20260924 16:15:30.295871 11984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c60caf4257f94463810f97475fd3499f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbb04676217f461282dc54deaf91a1a8"
I20260924 16:15:30.296165 11984 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cbb04676217f461282dc54deaf91a1a8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:15:30.299881 11984 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cbb04676217f461282dc54deaf91a1a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c60caf4257f94463810f97475fd3499f in term 1.
I20260924 16:15:30.300859 12097 leader_election.cc:288] T 00000000000000000000000000000000 P c60caf4257f94463810f97475fd3499f [CANDIDATE]: Term 1 election: Requested vote from peers 3390d4a9d3be457bb6854f3b63402763 (127.29.234.62:39487), cbb04676217f461282dc54deaf91a1a8 (127.29.234.61:42435)
I20260924 16:15:30.302937 12097 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c60caf4257f94463810f97475fd3499f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:15:30.303462 12097 raft_consensus.cc:686] T 00000000000000000000000000000000 P c60caf4257f94463810f97475fd3499f [term 1 LEADER]: Becoming Leader. State: Replica: c60caf4257f94463810f97475fd3499f, State: Running, Role: LEADER
I20260924 16:15:30.304503 12097 consensus_queue.cc:237] T 00000000000000000000000000000000 P c60caf4257f94463810f97475fd3499f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3390d4a9d3be457bb6854f3b63402763" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 39487 } } peers { permanent_uuid: "cbb04676217f461282dc54deaf91a1a8" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 42435 } } peers { permanent_uuid: "c60caf4257f94463810f97475fd3499f" member_type: VOTER last_known_addr { host: "127.29.234.60" port: 38803 } }
I20260924 16:15:30.315155 12138 sys_catalog.cc:455] T 00000000000000000000000000000000 P c60caf4257f94463810f97475fd3499f [sys.catalog]: SysCatalogTable state changed. Reason: New leader c60caf4257f94463810f97475fd3499f. Latest consensus state: current_term: 1 leader_uuid: "c60caf4257f94463810f97475fd3499f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3390d4a9d3be457bb6854f3b63402763" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 39487 } } peers { permanent_uuid: "cbb04676217f461282dc54deaf91a1a8" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 42435 } } peers { permanent_uuid: "c60caf4257f94463810f97475fd3499f" member_type: VOTER last_known_addr { host: "127.29.234.60" port: 38803 } } }
I20260924 16:15:30.315701 12138 sys_catalog.cc:458] T 00000000000000000000000000000000 P c60caf4257f94463810f97475fd3499f [sys.catalog]: This master's current role is: LEADER
I20260924 16:15:30.324635 12140 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:15:30.330598 12140 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:15:30.346573 11984 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cbb04676217f461282dc54deaf91a1a8 [term 1 FOLLOWER]: Refusing update from remote peer c60caf4257f94463810f97475fd3499f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:15:30.347877 12138 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c60caf4257f94463810f97475fd3499f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbb04676217f461282dc54deaf91a1a8" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 42435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:15:30.355175 11909 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3390d4a9d3be457bb6854f3b63402763 [term 1 FOLLOWER]: Refusing update from remote peer c60caf4257f94463810f97475fd3499f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:15:30.356098 12138 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c60caf4257f94463810f97475fd3499f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3390d4a9d3be457bb6854f3b63402763" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 39487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:15:30.375231 12111 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3390d4a9d3be457bb6854f3b63402763 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c60caf4257f94463810f97475fd3499f. Latest consensus state: current_term: 1 leader_uuid: "c60caf4257f94463810f97475fd3499f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3390d4a9d3be457bb6854f3b63402763" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 39487 } } peers { permanent_uuid: "cbb04676217f461282dc54deaf91a1a8" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 42435 } } peers { permanent_uuid: "c60caf4257f94463810f97475fd3499f" member_type: VOTER last_known_addr { host: "127.29.234.60" port: 38803 } } }
I20260924 16:15:30.375579 12111 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3390d4a9d3be457bb6854f3b63402763 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:15:30.379622 12141 mvcc.cc:205] Tried to move back new op lower bound from 7332931708286509056 to 7332931708140027904. Current Snapshot: MvccSnapshot[applied={T|T < 7332931708286509056}]
I20260924 16:15:30.383728 12097 sys_catalog.cc:455] T 00000000000000000000000000000000 P c60caf4257f94463810f97475fd3499f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c60caf4257f94463810f97475fd3499f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3390d4a9d3be457bb6854f3b63402763" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 39487 } } peers { permanent_uuid: "cbb04676217f461282dc54deaf91a1a8" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 42435 } } peers { permanent_uuid: "c60caf4257f94463810f97475fd3499f" member_type: VOTER last_known_addr { host: "127.29.234.60" port: 38803 } } }
I20260924 16:15:30.384048 12097 sys_catalog.cc:458] T 00000000000000000000000000000000 P c60caf4257f94463810f97475fd3499f [sys.catalog]: This master's current role is: LEADER
I20260924 16:15:30.385978 12140 catalog_manager.cc:1347] Generated new cluster ID: 71f4817af16b4522878fe1511b33cc9d
I20260924 16:15:30.386071 12140 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:15:30.387003 12125 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbb04676217f461282dc54deaf91a1a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c60caf4257f94463810f97475fd3499f. Latest consensus state: current_term: 1 leader_uuid: "c60caf4257f94463810f97475fd3499f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3390d4a9d3be457bb6854f3b63402763" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 39487 } } peers { permanent_uuid: "cbb04676217f461282dc54deaf91a1a8" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 42435 } } peers { permanent_uuid: "c60caf4257f94463810f97475fd3499f" member_type: VOTER last_known_addr { host: "127.29.234.60" port: 38803 } } }
I20260924 16:15:30.387334 12125 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbb04676217f461282dc54deaf91a1a8 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:15:30.399245 12138 sys_catalog.cc:455] T 00000000000000000000000000000000 P c60caf4257f94463810f97475fd3499f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c60caf4257f94463810f97475fd3499f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3390d4a9d3be457bb6854f3b63402763" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 39487 } } peers { permanent_uuid: "cbb04676217f461282dc54deaf91a1a8" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 42435 } } peers { permanent_uuid: "c60caf4257f94463810f97475fd3499f" member_type: VOTER last_known_addr { host: "127.29.234.60" port: 38803 } } }
I20260924 16:15:30.399657 12138 sys_catalog.cc:458] T 00000000000000000000000000000000 P c60caf4257f94463810f97475fd3499f [sys.catalog]: This master's current role is: LEADER
I20260924 16:15:30.403857 12111 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3390d4a9d3be457bb6854f3b63402763 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c60caf4257f94463810f97475fd3499f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3390d4a9d3be457bb6854f3b63402763" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 39487 } } peers { permanent_uuid: "cbb04676217f461282dc54deaf91a1a8" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 42435 } } peers { permanent_uuid: "c60caf4257f94463810f97475fd3499f" member_type: VOTER last_known_addr { host: "127.29.234.60" port: 38803 } } }
I20260924 16:15:30.404157 12111 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3390d4a9d3be457bb6854f3b63402763 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:15:30.408341 12125 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbb04676217f461282dc54deaf91a1a8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c60caf4257f94463810f97475fd3499f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3390d4a9d3be457bb6854f3b63402763" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 39487 } } peers { permanent_uuid: "cbb04676217f461282dc54deaf91a1a8" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 42435 } } peers { permanent_uuid: "c60caf4257f94463810f97475fd3499f" member_type: VOTER last_known_addr { host: "127.29.234.60" port: 38803 } } }
I20260924 16:15:30.408623 12125 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbb04676217f461282dc54deaf91a1a8 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:15:30.423244 12142 mvcc.cc:205] Tried to move back new op lower bound from 7332931708286509056 to 7332931708140027904. Current Snapshot: MvccSnapshot[applied={T|T < 7332931708286509056}]
I20260924 16:15:30.440418 12140 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:15:30.442101 12140 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:15:30.453303 12140 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c60caf4257f94463810f97475fd3499f: Generated new TSK 0
I20260924 16:15:30.454319 12140 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:15:30.963116 12122 catalog_manager.cc:1259] Loaded cluster ID: 71f4817af16b4522878fe1511b33cc9d
I20260924 16:15:30.963268 12122 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3390d4a9d3be457bb6854f3b63402763: loading cluster ID for follower catalog manager: success
I20260924 16:15:30.966406 12122 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3390d4a9d3be457bb6854f3b63402763: acquiring CA information for follower catalog manager: success
I20260924 16:15:30.968364 12122 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3390d4a9d3be457bb6854f3b63402763: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 16:15:31.056847 12136 catalog_manager.cc:1259] Loaded cluster ID: 71f4817af16b4522878fe1511b33cc9d
I20260924 16:15:31.056947 12136 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cbb04676217f461282dc54deaf91a1a8: loading cluster ID for follower catalog manager: success
I20260924 16:15:31.059500 12136 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cbb04676217f461282dc54deaf91a1a8: acquiring CA information for follower catalog manager: success
I20260924 16:15:31.061319 12136 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cbb04676217f461282dc54deaf91a1a8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 16:15:31.551717 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 11867
I20260924 16:15:31.553128 11930 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:15:31.664642 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 11867
I20260924 16:15:31.686898 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 11942
I20260924 16:15:31.688143 12004 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:15:31.796916 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 11942
I20260924 16:15:31.817988 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 12016
I20260924 16:15:31.819218 12079 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:15:31.931356 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 12016
2026-09-24T16:15:31Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4972 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790266323169505-30632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:15:32 dist-test-slave-v5j9 krb5kdc[12160](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:15:32 dist-test-slave-v5j9 krb5kdc[12160](info): set up 2 sockets
Sep 24 16:15:32 dist-test-slave-v5j9 krb5kdc[12160](info): commencing operation
krb5kdc: starting...
W20260924 16:15:33.037580 30632 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.005s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Sep 24 16:15:33 dist-test-slave-v5j9 krb5kdc[12160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:15:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:15:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.234.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab.
Entry for principal kudu/127.29.234.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab.
I20260924 16:15:33.276897 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790266323169505-30632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790266323169505-30632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790266323169505-30632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:39879
--webserver_interface=127.29.234.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab
--principal=kudu/127.29.234.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:37545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.234.62:39879
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:15:33.609278 12173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:15:33.609591 12173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:15:33.609802 12173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:15:33.616772 12173 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:15:33.616861 12173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:15:33.616966 12173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:15:33.617067 12173 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:15:33.617129 12173 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:15:33.626771 12173 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:37545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790266323169505-30632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790266323169505-30632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.234.62:39879
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab
--principal=kudu/127.29.234.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:39879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.234.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790266323169505-30632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:15:33.628144 12173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:15:33.629546 12173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:15:33.641636 12178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:33.641657 12179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:33.641724 12181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:15:33.642627 12173 server_base.cc:1034] running on GCE node
I20260924 16:15:33.643383 12173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:15:33.644805 12173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:15:33.646045 12173 hybrid_clock.cc:648] HybridClock initialized: now 1790266533645993 us; error 41 us; skew 500 ppm
I20260924 16:15:33.646440 12173 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:15:33 dist-test-slave-v5j9 krb5kdc[12160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266533, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:15:33.649791 12173 init.cc:377] Logged in from keytab as kudu/127.29.234.62@KRBTEST.COM (short username kudu)
I20260924 16:15:33.651465 12173 webserver.cc:460] Webserver started at http://127.29.234.62:43305/ using document root <none> and password file <none>
I20260924 16:15:33.652194 12173 fs_manager.cc:359] Metadata directory not provided
I20260924 16:15:33.652352 12173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:15:33.652675 12173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:15:33.655810 12173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790266323169505-30632-0/minicluster-data/master-0/data/instance:
uuid: "c7df157e61aa460e9e8dd4bc35f803b1"
format_stamp: "Formatted at 2026-09-24 16:15:33 on dist-test-slave-v5j9"
server_key: "0b41c400caabcea10f2d2699635ed05b"
server_key_iv: "bd4c4dbc8135aa62f7ab145faf10cf41"
server_key_version: "encryptionkey@0"
I20260924 16:15:33.656586 12173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790266323169505-30632-0/minicluster-data/master-0/wal/instance:
uuid: "c7df157e61aa460e9e8dd4bc35f803b1"
format_stamp: "Formatted at 2026-09-24 16:15:33 on dist-test-slave-v5j9"
server_key: "0b41c400caabcea10f2d2699635ed05b"
server_key_iv: "bd4c4dbc8135aa62f7ab145faf10cf41"
server_key_version: "encryptionkey@0"
I20260924 16:15:33.661964 12173 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 16:15:33.665520 12188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:33.666834 12173 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:15:33.667021 12173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790266323169505-30632-0/minicluster-data/master-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790266323169505-30632-0/minicluster-data/master-0/wal
uuid: "c7df157e61aa460e9e8dd4bc35f803b1"
format_stamp: "Formatted at 2026-09-24 16:15:33 on dist-test-slave-v5j9"
server_key: "0b41c400caabcea10f2d2699635ed05b"
server_key_iv: "bd4c4dbc8135aa62f7ab145faf10cf41"
server_key_version: "encryptionkey@0"
I20260924 16:15:33.667222 12173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790266323169505-30632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790266323169505-30632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790266323169505-30632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:15:33.681145 12173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:15:33.685392 12173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:15:33.685685 12173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:15:33.704905 12173 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.62:39879
I20260924 16:15:33.704911 12239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.62:39879 every 8 connection(s)
I20260924 16:15:33.706480 12173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
I20260924 16:15:33.709555 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 12173
I20260924 16:15:33.709802 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790266323169505-30632-0/minicluster-data/master-0/wal/instance
I20260924 16:15:33.710642 30632 external_mini_cluster.cc:1428] Setting key 216bee2ae081e48b25070cb34974fa71
I20260924 16:15:33.711061 12240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:15:33.725889 12240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7df157e61aa460e9e8dd4bc35f803b1: Bootstrap starting.
I20260924 16:15:33.729050 12240 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c7df157e61aa460e9e8dd4bc35f803b1: Neither blocks nor log segments found. Creating new log.
I20260924 16:15:33.730083 12240 log.cc:824] T 00000000000000000000000000000000 P c7df157e61aa460e9e8dd4bc35f803b1: Log is configured to *not* fsync() on all Append() calls
I20260924 16:15:33.733404 12240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7df157e61aa460e9e8dd4bc35f803b1: No bootstrap required, opened a new log
Sep 24 16:15:33 dist-test-slave-v5j9 krb5kdc[12160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:15:33.741720 12240 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7df157e61aa460e9e8dd4bc35f803b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7df157e61aa460e9e8dd4bc35f803b1" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 39879 } }
I20260924 16:15:33.742166 12240 raft_consensus.cc:374] T 00000000000000000000000000000000 P c7df157e61aa460e9e8dd4bc35f803b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:15:33.742283 12240 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7df157e61aa460e9e8dd4bc35f803b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7df157e61aa460e9e8dd4bc35f803b1, State: Initialized, Role: FOLLOWER
I20260924 16:15:33.742915 12240 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7df157e61aa460e9e8dd4bc35f803b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7df157e61aa460e9e8dd4bc35f803b1" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 39879 } }
I20260924 16:15:33.743155 12240 raft_consensus.cc:388] T 00000000000000000000000000000000 P c7df157e61aa460e9e8dd4bc35f803b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:15:33.743323 12240 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7df157e61aa460e9e8dd4bc35f803b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:15:33.743503 12240 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7df157e61aa460e9e8dd4bc35f803b1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:15:33.743685 12243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:33.712541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51584 (local address 127.29.234.62:39879)
0924 16:15:33.713328 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:33.713340 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:33.713440 (+ 100us) server_negotiation.cc:407] Connection header received
0924 16:15:33.715798 (+ 2358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:33.715825 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:33.716272 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:33.716641 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:33.718653 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:33.720000 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:33.721929 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:33.722669 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:33.722829 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:33.735526 (+ 12697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:33.735535 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:33.735554 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:15:33.735592 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:33.738854 (+ 3262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:33.739659 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:33.739667 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:33.739673 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:33.739863 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:33.740308 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:33.740312 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:33.740314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:33.740888 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:33.741194 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:33.741706 (+ 512us) server_negotiation.cc:299] Negotiation successful
0924 16:15:33.742277 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":370,"threads_started":1}
I20260924 16:15:33.746323 12240 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7df157e61aa460e9e8dd4bc35f803b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7df157e61aa460e9e8dd4bc35f803b1" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 39879 } }
I20260924 16:15:33.746920 12240 leader_election.cc:302] T 00000000000000000000000000000000 P c7df157e61aa460e9e8dd4bc35f803b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c7df157e61aa460e9e8dd4bc35f803b1; no voters:
I20260924 16:15:33.747562 12240 leader_election.cc:288] T 00000000000000000000000000000000 P c7df157e61aa460e9e8dd4bc35f803b1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:15:33.747717 12245 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c7df157e61aa460e9e8dd4bc35f803b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:15:33.747812 30632 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:15:33.750051 12240 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7df157e61aa460e9e8dd4bc35f803b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:15:33.751752 12245 raft_consensus.cc:686] T 00000000000000000000000000000000 P c7df157e61aa460e9e8dd4bc35f803b1 [term 1 LEADER]: Becoming Leader. State: Replica: c7df157e61aa460e9e8dd4bc35f803b1, State: Running, Role: LEADER
I20260924 16:15:33.752429 12245 consensus_queue.cc:237] T 00000000000000000000000000000000 P c7df157e61aa460e9e8dd4bc35f803b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7df157e61aa460e9e8dd4bc35f803b1" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 39879 } }
I20260924 16:15:33.755618 12247 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7df157e61aa460e9e8dd4bc35f803b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7df157e61aa460e9e8dd4bc35f803b1. Latest consensus state: current_term: 1 leader_uuid: "c7df157e61aa460e9e8dd4bc35f803b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7df157e61aa460e9e8dd4bc35f803b1" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 39879 } } }
I20260924 16:15:33.756052 12247 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7df157e61aa460e9e8dd4bc35f803b1 [sys.catalog]: This master's current role is: LEADER
I20260924 16:15:33.755693 12246 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7df157e61aa460e9e8dd4bc35f803b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c7df157e61aa460e9e8dd4bc35f803b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7df157e61aa460e9e8dd4bc35f803b1" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 39879 } } }
I20260924 16:15:33.756570 12246 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7df157e61aa460e9e8dd4bc35f803b1 [sys.catalog]: This master's current role is: LEADER
I20260924 16:15:33.757725 12256 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:15:33.761915 12256 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:15:33.770569 12256 catalog_manager.cc:1347] Generated new cluster ID: 464162ed14dc492393e683ea91ffeab1
I20260924 16:15:33.770694 12256 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:15:33.779397 12256 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:15:33.780634 12256 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:15:33.793041 12256 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c7df157e61aa460e9e8dd4bc35f803b1: Generated new TSK 0
I20260924 16:15:33.793946 12256 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:15:33.810890 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 12173
I20260924 16:15:33.812117 12235 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:15:33.909072 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 12173
2026-09-24T16:15:33Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1973 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:15:34 dist-test-slave-v5j9 krb5kdc[12269](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:15:34 dist-test-slave-v5j9 krb5kdc[12269](info): set up 2 sockets
Sep 24 16:15:34 dist-test-slave-v5j9 krb5kdc[12269](info): commencing operation
krb5kdc: starting...
W20260924 16:15:35.017544 30632 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.002s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Sep 24 16:15:35 dist-test-slave-v5j9 krb5kdc[12269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:15:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:15:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.234.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab.
Entry for principal kudu/127.29.234.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab.
I20260924 16:15:35.262586 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:35061
--webserver_interface=127.29.234.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab
--principal=kudu/127.29.234.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:45085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.234.62:35061
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:15:35.591446 12282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:15:35.591825 12282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:15:35.591995 12282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:15:35.599370 12282 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:15:35.599530 12282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:15:35.599638 12282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:15:35.599747 12282 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:15:35.599838 12282 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:15:35.609943 12282 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:45085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.234.62:35061
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab
--principal=kudu/127.29.234.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:35061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.234.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:15:35.611394 12282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:15:35.612731 12282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:15:35.624485 12288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:35.624428 12291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:35.624528 12287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:15:35.625317 12282 server_base.cc:1034] running on GCE node
I20260924 16:15:35.626068 12282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:15:35.627370 12282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:15:35.628655 12282 hybrid_clock.cc:648] HybridClock initialized: now 1790266535628586 us; error 58 us; skew 500 ppm
I20260924 16:15:35.629055 12282 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:15:35 dist-test-slave-v5j9 krb5kdc[12269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266535, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:15:35.632150 12282 init.cc:377] Logged in from keytab as kudu/127.29.234.62@KRBTEST.COM (short username kudu)
I20260924 16:15:35.633641 12282 webserver.cc:460] Webserver started at http://127.29.234.62:46067/ using document root <none> and password file <none>
I20260924 16:15:35.634405 12282 fs_manager.cc:359] Metadata directory not provided
I20260924 16:15:35.634568 12282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:15:35.634886 12282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:15:35.637903 12282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/master-0/data/instance:
uuid: "c32f836d405347ed9b2b10575a31a604"
format_stamp: "Formatted at 2026-09-24 16:15:35 on dist-test-slave-v5j9"
server_key: "534e502a9197cb96561efb4001391942"
server_key_iv: "3fe3b87a490785895b4bbba7ead71661"
server_key_version: "encryptionkey@0"
I20260924 16:15:35.638654 12282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/master-0/wal/instance:
uuid: "c32f836d405347ed9b2b10575a31a604"
format_stamp: "Formatted at 2026-09-24 16:15:35 on dist-test-slave-v5j9"
server_key: "534e502a9197cb96561efb4001391942"
server_key_iv: "3fe3b87a490785895b4bbba7ead71661"
server_key_version: "encryptionkey@0"
I20260924 16:15:35.643579 12282 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 16:15:35.647102 12298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:35.648670 12282 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 16:15:35.648864 12282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/master-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/master-0/wal
uuid: "c32f836d405347ed9b2b10575a31a604"
format_stamp: "Formatted at 2026-09-24 16:15:35 on dist-test-slave-v5j9"
server_key: "534e502a9197cb96561efb4001391942"
server_key_iv: "3fe3b87a490785895b4bbba7ead71661"
server_key_version: "encryptionkey@0"
I20260924 16:15:35.649071 12282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:15:35.678478 12282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:15:35.682754 12282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:15:35.683046 12282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:15:35.701874 12282 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.62:35061
I20260924 16:15:35.701897 12349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.62:35061 every 8 connection(s)
I20260924 16:15:35.703300 12282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
I20260924 16:15:35.705806 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 12282
I20260924 16:15:35.706029 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/master-0/wal/instance
I20260924 16:15:35.706734 30632 external_mini_cluster.cc:1428] Setting key 79647a00bbbde1bc7c34d16a2b133368
I20260924 16:15:35.707384 12350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:15:35.721832 12350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c32f836d405347ed9b2b10575a31a604: Bootstrap starting.
Sep 24 16:15:35 dist-test-slave-v5j9 krb5kdc[12269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:15:35.725716 12350 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c32f836d405347ed9b2b10575a31a604: Neither blocks nor log segments found. Creating new log.
I20260924 16:15:35.727087 12350 log.cc:824] T 00000000000000000000000000000000 P c32f836d405347ed9b2b10575a31a604: Log is configured to *not* fsync() on all Append() calls
I20260924 16:15:35.730559 12350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c32f836d405347ed9b2b10575a31a604: No bootstrap required, opened a new log
I20260924 16:15:35.731974 12353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:35.708823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36718 (local address 127.29.234.62:35061)
0924 16:15:35.709658 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:35.709673 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:35.709732 (+ 59us) server_negotiation.cc:407] Connection header received
0924 16:15:35.712221 (+ 2489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:35.712257 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:35.712819 (+ 562us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:35.713271 (+ 452us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:35.714652 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:35.716629 (+ 1977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:35.718012 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:35.719133 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:35.719312 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:35.723376 (+ 4064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:35.723388 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:35.723407 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:15:35.723467 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:35.727441 (+ 3974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:35.728299 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:35.728305 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:35.728314 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:35.728514 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:35.728881 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:35.728885 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:35.728888 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:35.729601 (+ 713us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:35.729936 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:35.730208 (+ 272us) server_negotiation.cc:299] Negotiation successful
0924 16:15:35.730646 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":318,"threads_started":1}
I20260924 16:15:35.739058 12350 raft_consensus.cc:348] T 00000000000000000000000000000000 P c32f836d405347ed9b2b10575a31a604 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c32f836d405347ed9b2b10575a31a604" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 35061 } }
I20260924 16:15:35.739425 12350 raft_consensus.cc:374] T 00000000000000000000000000000000 P c32f836d405347ed9b2b10575a31a604 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:15:35.739506 12350 raft_consensus.cc:729] T 00000000000000000000000000000000 P c32f836d405347ed9b2b10575a31a604 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c32f836d405347ed9b2b10575a31a604, State: Initialized, Role: FOLLOWER
I20260924 16:15:35.739982 12350 consensus_queue.cc:260] T 00000000000000000000000000000000 P c32f836d405347ed9b2b10575a31a604 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c32f836d405347ed9b2b10575a31a604" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 35061 } }
I20260924 16:15:35.740180 12350 raft_consensus.cc:388] T 00000000000000000000000000000000 P c32f836d405347ed9b2b10575a31a604 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:15:35.740288 12350 raft_consensus.cc:482] T 00000000000000000000000000000000 P c32f836d405347ed9b2b10575a31a604 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:15:35.740407 12350 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c32f836d405347ed9b2b10575a31a604 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:15:35.742784 12350 raft_consensus.cc:504] T 00000000000000000000000000000000 P c32f836d405347ed9b2b10575a31a604 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c32f836d405347ed9b2b10575a31a604" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 35061 } }
I20260924 16:15:35.743233 12350 leader_election.cc:302] T 00000000000000000000000000000000 P c32f836d405347ed9b2b10575a31a604 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c32f836d405347ed9b2b10575a31a604; no voters:
I20260924 16:15:35.743752 12350 leader_election.cc:288] T 00000000000000000000000000000000 P c32f836d405347ed9b2b10575a31a604 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:15:35.743914 12355 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c32f836d405347ed9b2b10575a31a604 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:15:35.744630 12355 raft_consensus.cc:686] T 00000000000000000000000000000000 P c32f836d405347ed9b2b10575a31a604 [term 1 LEADER]: Becoming Leader. State: Replica: c32f836d405347ed9b2b10575a31a604, State: Running, Role: LEADER
I20260924 16:15:35.745182 12355 consensus_queue.cc:237] T 00000000000000000000000000000000 P c32f836d405347ed9b2b10575a31a604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c32f836d405347ed9b2b10575a31a604" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 35061 } }
I20260924 16:15:35.745909 12350 sys_catalog.cc:564] T 00000000000000000000000000000000 P c32f836d405347ed9b2b10575a31a604 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:15:35.748266 12356 sys_catalog.cc:455] T 00000000000000000000000000000000 P c32f836d405347ed9b2b10575a31a604 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c32f836d405347ed9b2b10575a31a604" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c32f836d405347ed9b2b10575a31a604" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 35061 } } }
I20260924 16:15:35.748919 12356 sys_catalog.cc:458] T 00000000000000000000000000000000 P c32f836d405347ed9b2b10575a31a604 [sys.catalog]: This master's current role is: LEADER
I20260924 16:15:35.750406 12357 sys_catalog.cc:455] T 00000000000000000000000000000000 P c32f836d405347ed9b2b10575a31a604 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c32f836d405347ed9b2b10575a31a604. Latest consensus state: current_term: 1 leader_uuid: "c32f836d405347ed9b2b10575a31a604" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c32f836d405347ed9b2b10575a31a604" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 35061 } } }
I20260924 16:15:35.750876 12357 sys_catalog.cc:458] T 00000000000000000000000000000000 P c32f836d405347ed9b2b10575a31a604 [sys.catalog]: This master's current role is: LEADER
I20260924 16:15:35.754531 12363 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:15:35.760102 12363 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:15:35.769209 12363 catalog_manager.cc:1347] Generated new cluster ID: 5f4cda6912f7450594aa256ddbe1ceb6
I20260924 16:15:35.769309 12363 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:15:35.789434 12363 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:15:35.790802 12363 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:15:35.802498 12363 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c32f836d405347ed9b2b10575a31a604: Generated new TSK 0
I20260924 16:15:35.803357 12363 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.234.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab.
Entry for principal kudu/127.29.234.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab.
I20260924 16:15:35.882035 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.1:0
--local_ip_for_outbound_sockets=127.29.234.1
--webserver_interface=127.29.234.1
--webserver_port=0
--tserver_master_addrs=127.29.234.62:35061
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab
--principal=kudu/127.29.234.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:45085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:15:36.218866 12377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:15:36.219120 12377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:15:36.219285 12377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:15:36.226334 12377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:15:36.226436 12377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:15:36.226610 12377 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.1
I20260924 16:15:36.236315 12377 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:45085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab
--principal=kudu/127.29.234.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.234.1
--webserver_port=0
--tserver_master_addrs=127.29.234.62:35061
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.1
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:15:36.237740 12377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:15:36.239063 12377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:15:36.252072 12382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:36.252244 12383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:36.252108 12385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:15:36.253227 12377 server_base.cc:1034] running on GCE node
I20260924 16:15:36.253999 12377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:15:36.254945 12377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:15:36.256206 12377 hybrid_clock.cc:648] HybridClock initialized: now 1790266536256134 us; error 59 us; skew 500 ppm
I20260924 16:15:36.256528 12377 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:15:36 dist-test-slave-v5j9 krb5kdc[12269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266536, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:15:36.259728 12377 init.cc:377] Logged in from keytab as kudu/127.29.234.1@KRBTEST.COM (short username kudu)
I20260924 16:15:36.261138 12377 webserver.cc:460] Webserver started at http://127.29.234.1:40655/ using document root <none> and password file <none>
I20260924 16:15:36.261888 12377 fs_manager.cc:359] Metadata directory not provided
I20260924 16:15:36.262043 12377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:15:36.262353 12377 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:15:36.265300 12377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-0/data/instance:
uuid: "f991653d74f04af180776405e08e2bd0"
format_stamp: "Formatted at 2026-09-24 16:15:36 on dist-test-slave-v5j9"
server_key: "c8b280f8598c43a400bc4fdeb8e2648f"
server_key_iv: "0d2a977e2b1d6e86b92f43b018962069"
server_key_version: "encryptionkey@0"
I20260924 16:15:36.266185 12377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-0/wal/instance:
uuid: "f991653d74f04af180776405e08e2bd0"
format_stamp: "Formatted at 2026-09-24 16:15:36 on dist-test-slave-v5j9"
server_key: "c8b280f8598c43a400bc4fdeb8e2648f"
server_key_iv: "0d2a977e2b1d6e86b92f43b018962069"
server_key_version: "encryptionkey@0"
I20260924 16:15:36.271584 12377 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 16:15:36.275208 12392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:36.276717 12377 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 16:15:36.276901 12377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-0/wal
uuid: "f991653d74f04af180776405e08e2bd0"
format_stamp: "Formatted at 2026-09-24 16:15:36 on dist-test-slave-v5j9"
server_key: "c8b280f8598c43a400bc4fdeb8e2648f"
server_key_iv: "0d2a977e2b1d6e86b92f43b018962069"
server_key_version: "encryptionkey@0"
I20260924 16:15:36.277107 12377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:15:36.299556 12377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:15:36.304034 12377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:15:36.304342 12377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:15:36.305392 12377 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:15:36.306759 12377 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:15:36.306833 12377 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:36.306989 12377 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:15:36.307049 12377 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:36.338900 12377 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.1:42789
I20260924 16:15:36.338989 12507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.1:42789 every 8 connection(s)
I20260924 16:15:36.340238 12377 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
I20260924 16:15:36.345717 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 12377
I20260924 16:15:36.345940 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-0/wal/instance
I20260924 16:15:36.346629 30632 external_mini_cluster.cc:1428] Setting key e298aad273a6698e2a9665f492c84ea5
Sep 24 16:15:36 dist-test-slave-v5j9 krb5kdc[12269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266536, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.1@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:15:36.364596 12510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:36.343094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.1:35345 (local address 127.29.234.62:35061)
0924 16:15:36.343545 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:36.343549 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:36.344534 (+ 985us) server_negotiation.cc:407] Connection header received
0924 16:15:36.346603 (+ 2069us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:36.346609 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:36.346753 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:36.346888 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:36.348859 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:36.349656 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:36.352251 (+ 2595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:36.352637 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:36.352794 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:36.357622 (+ 4828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:36.357630 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:36.357638 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:15:36.357684 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:36.361303 (+ 3619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:36.362044 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:36.362052 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:36.362058 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:36.362228 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:36.362628 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:36.362635 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:36.362641 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:36.363013 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:36.363173 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:36.363832 (+ 659us) server_negotiation.cc:299] Negotiation successful
0924 16:15:36.364088 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":215,"threads_started":1}
I20260924 16:15:36.365306 12511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:36.343686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:35061 (local address 127.29.234.1:35345)
0924 16:15:36.344365 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:36.344409 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:36.346303 (+ 1894us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:36.347179 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:36.347195 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:36.347935 (+ 740us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:15:36.348642 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:36.348691 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:36.349823 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:36.349832 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:36.352067 (+ 2235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:36.352086 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:36.352897 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:36.352907 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:36.353239 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:36.354126 (+ 887us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:15:36.354161 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:15:36.357363 (+ 3202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:15:36.361473 (+ 4110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:15:36.361480 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:15:36.361495 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:15:36.361917 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:15:36.362341 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:15:36.362345 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:15:36.362347 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:15:36.362522 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:15:36.363132 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:15:36.363137 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:15:36.363453 (+ 316us) client_negotiation.cc:769] Sending connection context
0924 16:15:36.363679 (+ 226us) client_negotiation.cc:240] Negotiation successful
0924 16:15:36.364071 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":244,"threads_started":1}
I20260924 16:15:36.366971 12508 heartbeater.cc:347] Connected to a master server at 127.29.234.62:35061
I20260924 16:15:36.367317 12508 heartbeater.cc:464] Registering TS with master...
I20260924 16:15:36.368059 12508 heartbeater.cc:511] Master 127.29.234.62:35061 requested a full tablet report, sending...
I20260924 16:15:36.370318 12315 ts_manager.cc:194] Registered new tserver with Master: f991653d74f04af180776405e08e2bd0 (127.29.234.1:42789)
I20260924 16:15:36.371757 12315 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.234.1@KRBTEST.COM'} at 127.29.234.1:35345
WARNING: no policy specified for kudu/127.29.234.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab.
Entry for principal kudu/127.29.234.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab.
I20260924 16:15:36.401067 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.2:0
--local_ip_for_outbound_sockets=127.29.234.2
--webserver_interface=127.29.234.2
--webserver_port=0
--tserver_master_addrs=127.29.234.62:35061
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab
--principal=kudu/127.29.234.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:45085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:15:36.732460 12514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:15:36.732748 12514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:15:36.732928 12514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:15:36.740324 12514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:15:36.740434 12514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:15:36.740621 12514 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.2
I20260924 16:15:36.750767 12514 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:45085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab
--principal=kudu/127.29.234.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.234.2
--webserver_port=0
--tserver_master_addrs=127.29.234.62:35061
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.2
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:15:36.752106 12514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:15:36.753746 12514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:15:36.767787 12520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:36.767750 12522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:36.767822 12519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:15:36.768479 12514 server_base.cc:1034] running on GCE node
I20260924 16:15:36.769104 12514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:15:36.770110 12514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:15:36.771311 12514 hybrid_clock.cc:648] HybridClock initialized: now 1790266536771266 us; error 33 us; skew 500 ppm
I20260924 16:15:36.771636 12514 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:15:36 dist-test-slave-v5j9 krb5kdc[12269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266536, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:15:36.775003 12514 init.cc:377] Logged in from keytab as kudu/127.29.234.2@KRBTEST.COM (short username kudu)
I20260924 16:15:36.777006 12514 webserver.cc:460] Webserver started at http://127.29.234.2:44075/ using document root <none> and password file <none>
I20260924 16:15:36.777786 12514 fs_manager.cc:359] Metadata directory not provided
I20260924 16:15:36.777932 12514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:15:36.778239 12514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:15:36.781275 12514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-1/data/instance:
uuid: "493291c4622e44f19e482afeb9191500"
format_stamp: "Formatted at 2026-09-24 16:15:36 on dist-test-slave-v5j9"
server_key: "277fe4ba9eb1034086a741e02efada08"
server_key_iv: "a93b1f0c7143d992a7e5f8fb6e984152"
server_key_version: "encryptionkey@0"
I20260924 16:15:36.782261 12514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-1/wal/instance:
uuid: "493291c4622e44f19e482afeb9191500"
format_stamp: "Formatted at 2026-09-24 16:15:36 on dist-test-slave-v5j9"
server_key: "277fe4ba9eb1034086a741e02efada08"
server_key_iv: "a93b1f0c7143d992a7e5f8fb6e984152"
server_key_version: "encryptionkey@0"
I20260924 16:15:36.787458 12514 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260924 16:15:36.790944 12529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:36.792368 12514 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 16:15:36.792549 12514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-1/wal
uuid: "493291c4622e44f19e482afeb9191500"
format_stamp: "Formatted at 2026-09-24 16:15:36 on dist-test-slave-v5j9"
server_key: "277fe4ba9eb1034086a741e02efada08"
server_key_iv: "a93b1f0c7143d992a7e5f8fb6e984152"
server_key_version: "encryptionkey@0"
I20260924 16:15:36.792761 12514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:15:36.806792 12514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:15:36.810640 12514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:15:36.810927 12514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:15:36.811902 12514 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:15:36.813227 12514 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:15:36.813329 12514 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:36.813480 12514 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:15:36.813558 12514 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:36.846789 12514 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.2:41577
I20260924 16:15:36.846830 12644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.2:41577 every 8 connection(s)
I20260924 16:15:36.848206 12514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
I20260924 16:15:36.852927 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 12514
I20260924 16:15:36.853133 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-1/wal/instance
I20260924 16:15:36.853716 30632 external_mini_cluster.cc:1428] Setting key 0d55ce90b49b296aac8d6bca04d0f022
Sep 24 16:15:36 dist-test-slave-v5j9 krb5kdc[12269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266536, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.2@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:15:36.870122 12510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:36.851008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.2:41559 (local address 127.29.234.62:35061)
0924 16:15:36.851206 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:36.851210 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:36.852632 (+ 1422us) server_negotiation.cc:407] Connection header received
0924 16:15:36.854848 (+ 2216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:36.854854 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:36.854951 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:36.855091 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:36.856869 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:36.857630 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:36.859197 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:36.859614 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:36.859760 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:36.863936 (+ 4176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:36.863941 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:36.863944 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:15:36.863979 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:36.866711 (+ 2732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:36.867673 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:36.867677 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:36.867679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:36.867805 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:36.868320 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:36.868323 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:36.868325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:36.868589 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:36.868710 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:36.869679 (+ 969us) server_negotiation.cc:299] Negotiation successful
0924 16:15:36.869870 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":73}
I20260924 16:15:36.871722 12647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:36.851621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:35061 (local address 127.29.234.2:41559)
0924 16:15:36.852476 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:36.852514 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:36.854570 (+ 2056us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:36.855480 (+ 910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:36.855489 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:36.856079 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:15:36.856696 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:36.856717 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:36.857789 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:36.857802 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:36.859050 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:36.859064 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:36.859854 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:36.859861 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:36.860104 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:36.860828 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:15:36.860851 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:15:36.863689 (+ 2838us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:15:36.866864 (+ 3175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:15:36.866876 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:15:36.866897 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:15:36.867544 (+ 647us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:15:36.867917 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:15:36.867925 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:15:36.867931 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:15:36.868194 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:15:36.868719 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:15:36.868730 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:15:36.869240 (+ 510us) client_negotiation.cc:769] Sending connection context
0924 16:15:36.869594 (+ 354us) client_negotiation.cc:240] Negotiation successful
0924 16:15:36.870217 (+ 623us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":337,"threads_started":1}
I20260924 16:15:36.873281 12645 heartbeater.cc:347] Connected to a master server at 127.29.234.62:35061
I20260924 16:15:36.873662 12645 heartbeater.cc:464] Registering TS with master...
I20260924 16:15:36.874632 12645 heartbeater.cc:511] Master 127.29.234.62:35061 requested a full tablet report, sending...
I20260924 16:15:36.876309 12315 ts_manager.cc:194] Registered new tserver with Master: 493291c4622e44f19e482afeb9191500 (127.29.234.2:41577)
I20260924 16:15:36.877148 12315 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.234.2@KRBTEST.COM'} at 127.29.234.2:41559
WARNING: no policy specified for kudu/127.29.234.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab.
Entry for principal kudu/127.29.234.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab.
I20260924 16:15:36.908932 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.3:0
--local_ip_for_outbound_sockets=127.29.234.3
--webserver_interface=127.29.234.3
--webserver_port=0
--tserver_master_addrs=127.29.234.62:35061
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab
--principal=kudu/127.29.234.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:45085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:15:37.243950 12650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:15:37.244319 12650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:15:37.244490 12650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:15:37.252223 12650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:15:37.252419 12650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:15:37.252600 12650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.3
I20260924 16:15:37.262887 12650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:45085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab
--principal=kudu/127.29.234.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.234.3
--webserver_port=0
--tserver_master_addrs=127.29.234.62:35061
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.3
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:15:37.264273 12650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:15:37.265799 12650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:15:37.279098 12655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:37.279042 12658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:37.279156 12656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:15:37.280030 12650 server_base.cc:1034] running on GCE node
I20260924 16:15:37.280607 12650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:15:37.281579 12650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:15:37.282809 12650 hybrid_clock.cc:648] HybridClock initialized: now 1790266537282765 us; error 36 us; skew 500 ppm
I20260924 16:15:37.283135 12650 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:15:37 dist-test-slave-v5j9 krb5kdc[12269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266537, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:15:37.286347 12650 init.cc:377] Logged in from keytab as kudu/127.29.234.3@KRBTEST.COM (short username kudu)
I20260924 16:15:37.287755 12650 webserver.cc:460] Webserver started at http://127.29.234.3:44797/ using document root <none> and password file <none>
I20260924 16:15:37.288443 12650 fs_manager.cc:359] Metadata directory not provided
I20260924 16:15:37.288594 12650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:15:37.288904 12650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:15:37.291905 12650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-2/data/instance:
uuid: "12899fec5f5c4194af251b18585e0469"
format_stamp: "Formatted at 2026-09-24 16:15:37 on dist-test-slave-v5j9"
server_key: "6754491885e551fa0c9efa254bbfbcd9"
server_key_iv: "d977910b03c2242e1b2b3f4495f600b1"
server_key_version: "encryptionkey@0"
I20260924 16:15:37.292711 12650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-2/wal/instance:
uuid: "12899fec5f5c4194af251b18585e0469"
format_stamp: "Formatted at 2026-09-24 16:15:37 on dist-test-slave-v5j9"
server_key: "6754491885e551fa0c9efa254bbfbcd9"
server_key_iv: "d977910b03c2242e1b2b3f4495f600b1"
server_key_version: "encryptionkey@0"
I20260924 16:15:37.297987 12650 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260924 16:15:37.301771 12665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:37.303203 12650 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:15:37.303376 12650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-2/wal
uuid: "12899fec5f5c4194af251b18585e0469"
format_stamp: "Formatted at 2026-09-24 16:15:37 on dist-test-slave-v5j9"
server_key: "6754491885e551fa0c9efa254bbfbcd9"
server_key_iv: "d977910b03c2242e1b2b3f4495f600b1"
server_key_version: "encryptionkey@0"
I20260924 16:15:37.303572 12650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:15:37.320734 12650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:15:37.324445 12650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:15:37.324733 12650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:15:37.325664 12650 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:15:37.326965 12650 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:15:37.327070 12650 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:37.327219 12650 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:15:37.327296 12650 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:37.360946 12650 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.3:38119
I20260924 16:15:37.360991 12779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.3:38119 every 8 connection(s)
I20260924 16:15:37.362385 12650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
I20260924 16:15:37.372140 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 12650
I20260924 16:15:37.372334 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-2/wal/instance
I20260924 16:15:37.372929 30632 external_mini_cluster.cc:1428] Setting key 4d7e6332afcf7bd026b4d00f619596f3
I20260924 16:15:37.375092 12508 heartbeater.cc:503] Master 127.29.234.62:35061 was elected leader, sending a full tablet report...
Sep 24 16:15:37 dist-test-slave-v5j9 krb5kdc[12269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266537, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.3@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:15:37.383306 12510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:37.365364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.3:52837 (local address 127.29.234.62:35061)
0924 16:15:37.365544 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:37.365549 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:37.366663 (+ 1114us) server_negotiation.cc:407] Connection header received
0924 16:15:37.368435 (+ 1772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:37.368440 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:37.368532 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:37.368643 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:37.370168 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:37.370922 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:37.372919 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:37.373430 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:37.373587 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:37.377765 (+ 4178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:37.377772 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:37.377782 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:15:37.377814 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:37.380312 (+ 2498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:37.380989 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:37.380996 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:37.381004 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:37.381165 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:37.381632 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:37.381637 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:37.381640 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:37.381893 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:37.382085 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:37.382749 (+ 664us) server_negotiation.cc:299] Negotiation successful
0924 16:15:37.382960 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 16:15:37.384213 12782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:37.365811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:35061 (local address 127.29.234.3:52837)
0924 16:15:37.366508 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:37.366551 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:37.368171 (+ 1620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:37.368880 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:37.368888 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:37.369453 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:15:37.370001 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:37.370017 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:37.371082 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:37.371086 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:37.372756 (+ 1670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:37.372772 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:37.373660 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:37.373665 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:37.373977 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:37.374752 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:15:37.374817 (+ 65us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:15:37.377547 (+ 2730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:15:37.380439 (+ 2892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:15:37.380447 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:15:37.380463 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:15:37.380861 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:15:37.381279 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:15:37.381285 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:15:37.381288 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:15:37.381519 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:15:37.382043 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:15:37.382053 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:15:37.382389 (+ 336us) client_negotiation.cc:769] Sending connection context
0924 16:15:37.382640 (+ 251us) client_negotiation.cc:240] Negotiation successful
0924 16:15:37.383128 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":267,"threads_started":1}
I20260924 16:15:37.385627 12780 heartbeater.cc:347] Connected to a master server at 127.29.234.62:35061
I20260924 16:15:37.385929 12780 heartbeater.cc:464] Registering TS with master...
I20260924 16:15:37.386683 12780 heartbeater.cc:511] Master 127.29.234.62:35061 requested a full tablet report, sending...
I20260924 16:15:37.388356 12314 ts_manager.cc:194] Registered new tserver with Master: 12899fec5f5c4194af251b18585e0469 (127.29.234.3:38119)
I20260924 16:15:37.388914 30632 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:15:37.389557 12314 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.234.3@KRBTEST.COM'} at 127.29.234.3:52837
I20260924 16:15:37.406756 12510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:37.393748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36734 (local address 127.29.234.62:35061)
0924 16:15:37.393978 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:37.393988 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:37.394351 (+ 363us) server_negotiation.cc:407] Connection header received
0924 16:15:37.394511 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:37.394515 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:37.394579 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:37.394675 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:37.396040 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:37.396874 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:37.398482 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:37.398965 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:37.399120 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:37.401533 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:37.401541 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:37.401551 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:15:37.401585 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:37.404050 (+ 2465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:37.404733 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:37.404739 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:37.404743 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:37.404889 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:37.405271 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:37.405276 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:37.405279 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:37.405652 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:37.405908 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:37.406230 (+ 322us) server_negotiation.cc:299] Negotiation successful
0924 16:15:37.406443 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 16:15:37.410780 12314 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36734:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:15:37.417577 12314 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:15:37.453882 12795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:37.430889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46858 (local address 127.29.234.2:41577)
0924 16:15:37.445677 (+ 14788us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:37.445687 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:37.445721 (+ 34us) server_negotiation.cc:407] Connection header received
0924 16:15:37.445831 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:37.445838 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:37.446064 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:37.446280 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:15:37.447317 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:37.448781 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:37.451203 (+ 2422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:37.452606 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:37.452762 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:37.452959 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:37.453333 (+ 374us) server_negotiation.cc:299] Negotiation successful
0924 16:15:37.453542 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14658,"thread_start_us":14539,"threads_started":1}
I20260924 16:15:37.454054 12792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:37.430305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.2:41577 (local address 127.0.0.1:46858)
0924 16:15:37.444770 (+ 14465us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:37.444800 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:37.444940 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:37.446496 (+ 1556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:37.446500 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:37.446522 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:15:37.447079 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:37.447093 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:37.448933 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:37.448942 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:37.451060 (+ 2118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:37.451073 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:37.452804 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:37.452810 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:37.453039 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:37.453108 (+ 69us) client_negotiation.cc:769] Sending connection context
0924 16:15:37.453262 (+ 154us) client_negotiation.cc:240] Negotiation successful
0924 16:15:37.453498 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14267,"mutex_wait_us":45,"thread_start_us":769,"threads_started":1}
I20260924 16:15:37.454262 12794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:37.429881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:42789 (local address 127.0.0.1:52964)
0924 16:15:37.431345 (+ 1464us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:37.431384 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:37.431561 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:37.444126 (+ 12565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:37.444137 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:37.444181 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:15:37.444735 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:37.444756 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:37.447743 (+ 2987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:37.447749 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:37.450773 (+ 3024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:37.450788 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:37.452982 (+ 2194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:37.452988 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:37.453244 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:37.453278 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 16:15:37.453401 (+ 123us) client_negotiation.cc:240] Negotiation successful
0924 16:15:37.453542 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1211,"mutex_wait_us":118,"thread_start_us":15151,"threads_started":1}
I20260924 16:15:37.454641 12793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:37.430154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52964 (local address 127.29.234.1:42789)
0924 16:15:37.443313 (+ 13159us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:37.443327 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:37.443368 (+ 41us) server_negotiation.cc:407] Connection header received
0924 16:15:37.443496 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:37.443507 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:37.443849 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:37.444083 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:15:37.444994 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:37.447596 (+ 2602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:37.450934 (+ 3338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:37.452386 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:37.452538 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:37.452842 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:37.454059 (+ 1217us) server_negotiation.cc:299] Negotiation successful
0924 16:15:37.454226 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12920,"thread_start_us":14162,"threads_started":1}
I20260924 16:15:37.455938 12797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:37.432624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51568 (local address 127.29.234.3:38119)
0924 16:15:37.447141 (+ 14517us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:37.447150 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:37.447805 (+ 655us) server_negotiation.cc:407] Connection header received
0924 16:15:37.447910 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:37.447919 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:37.448131 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:37.448313 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:15:37.449215 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:37.450283 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:37.452672 (+ 2389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:37.454240 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:37.454683 (+ 443us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:37.454924 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:37.455428 (+ 504us) server_negotiation.cc:299] Negotiation successful
0924 16:15:37.455590 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14331,"thread_start_us":14177,"threads_started":1}
I20260924 16:15:37.456610 12796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:37.431914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.3:38119 (local address 127.0.0.1:51568)
0924 16:15:37.447592 (+ 15678us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:37.447611 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:37.447733 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:37.448325 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:37.448331 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:37.448355 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:15:37.449038 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:37.449052 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:37.450414 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:37.450419 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:37.452497 (+ 2078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:37.452511 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:37.455056 (+ 2545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:37.455063 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:37.455237 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:37.455267 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 16:15:37.455367 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 16:15:37.455490 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15526,"mutex_wait_us":232,"thread_start_us":15366,"threads_started":1}
I20260924 16:15:37.458401 12578 tablet_service.cc:1467] Processing CreateTablet for tablet 88626003aab141af8bb168b8f0454bb7 (DEFAULT_TABLE table=test-table [id=407fe6ae8f3241b496968694d5bbe06a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:15:37.458451 12440 tablet_service.cc:1467] Processing CreateTablet for tablet 88626003aab141af8bb168b8f0454bb7 (DEFAULT_TABLE table=test-table [id=407fe6ae8f3241b496968694d5bbe06a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:15:37.459830 12578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88626003aab141af8bb168b8f0454bb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:15:37.459990 12440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88626003aab141af8bb168b8f0454bb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:15:37.460309 12713 tablet_service.cc:1467] Processing CreateTablet for tablet 88626003aab141af8bb168b8f0454bb7 (DEFAULT_TABLE table=test-table [id=407fe6ae8f3241b496968694d5bbe06a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:15:37.461647 12713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88626003aab141af8bb168b8f0454bb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:15:37.470863 12798 tablet_bootstrap.cc:492] T 88626003aab141af8bb168b8f0454bb7 P 493291c4622e44f19e482afeb9191500: Bootstrap starting.
I20260924 16:15:37.474392 12798 tablet_bootstrap.cc:654] T 88626003aab141af8bb168b8f0454bb7 P 493291c4622e44f19e482afeb9191500: Neither blocks nor log segments found. Creating new log.
I20260924 16:15:37.474491 12799 tablet_bootstrap.cc:492] T 88626003aab141af8bb168b8f0454bb7 P f991653d74f04af180776405e08e2bd0: Bootstrap starting.
I20260924 16:15:37.475804 12798 log.cc:824] T 88626003aab141af8bb168b8f0454bb7 P 493291c4622e44f19e482afeb9191500: Log is configured to *not* fsync() on all Append() calls
I20260924 16:15:37.476094 12800 tablet_bootstrap.cc:492] T 88626003aab141af8bb168b8f0454bb7 P 12899fec5f5c4194af251b18585e0469: Bootstrap starting.
I20260924 16:15:37.478603 12799 tablet_bootstrap.cc:654] T 88626003aab141af8bb168b8f0454bb7 P f991653d74f04af180776405e08e2bd0: Neither blocks nor log segments found. Creating new log.
I20260924 16:15:37.479086 12800 tablet_bootstrap.cc:654] T 88626003aab141af8bb168b8f0454bb7 P 12899fec5f5c4194af251b18585e0469: Neither blocks nor log segments found. Creating new log.
I20260924 16:15:37.480252 12800 log.cc:824] T 88626003aab141af8bb168b8f0454bb7 P 12899fec5f5c4194af251b18585e0469: Log is configured to *not* fsync() on all Append() calls
I20260924 16:15:37.480223 12799 log.cc:824] T 88626003aab141af8bb168b8f0454bb7 P f991653d74f04af180776405e08e2bd0: Log is configured to *not* fsync() on all Append() calls
I20260924 16:15:37.483028 12800 tablet_bootstrap.cc:492] T 88626003aab141af8bb168b8f0454bb7 P 12899fec5f5c4194af251b18585e0469: No bootstrap required, opened a new log
I20260924 16:15:37.483304 12800 ts_tablet_manager.cc:1397] T 88626003aab141af8bb168b8f0454bb7 P 12899fec5f5c4194af251b18585e0469: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260924 16:15:37.485034 12799 tablet_bootstrap.cc:492] T 88626003aab141af8bb168b8f0454bb7 P f991653d74f04af180776405e08e2bd0: No bootstrap required, opened a new log
I20260924 16:15:37.485121 12798 tablet_bootstrap.cc:492] T 88626003aab141af8bb168b8f0454bb7 P 493291c4622e44f19e482afeb9191500: No bootstrap required, opened a new log
I20260924 16:15:37.485450 12798 ts_tablet_manager.cc:1397] T 88626003aab141af8bb168b8f0454bb7 P 493291c4622e44f19e482afeb9191500: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.004s
I20260924 16:15:37.485471 12799 ts_tablet_manager.cc:1397] T 88626003aab141af8bb168b8f0454bb7 P f991653d74f04af180776405e08e2bd0: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20260924 16:15:37.493068 12799 raft_consensus.cc:348] T 88626003aab141af8bb168b8f0454bb7 P f991653d74f04af180776405e08e2bd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f991653d74f04af180776405e08e2bd0" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 42789 } } peers { permanent_uuid: "493291c4622e44f19e482afeb9191500" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41577 } } peers { permanent_uuid: "12899fec5f5c4194af251b18585e0469" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 38119 } }
I20260924 16:15:37.493472 12799 raft_consensus.cc:374] T 88626003aab141af8bb168b8f0454bb7 P f991653d74f04af180776405e08e2bd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:15:37.493573 12799 raft_consensus.cc:729] T 88626003aab141af8bb168b8f0454bb7 P f991653d74f04af180776405e08e2bd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f991653d74f04af180776405e08e2bd0, State: Initialized, Role: FOLLOWER
I20260924 16:15:37.494220 12799 consensus_queue.cc:260] T 88626003aab141af8bb168b8f0454bb7 P f991653d74f04af180776405e08e2bd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f991653d74f04af180776405e08e2bd0" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 42789 } } peers { permanent_uuid: "493291c4622e44f19e482afeb9191500" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41577 } } peers { permanent_uuid: "12899fec5f5c4194af251b18585e0469" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 38119 } }
I20260924 16:15:37.494275 12800 raft_consensus.cc:348] T 88626003aab141af8bb168b8f0454bb7 P 12899fec5f5c4194af251b18585e0469 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f991653d74f04af180776405e08e2bd0" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 42789 } } peers { permanent_uuid: "493291c4622e44f19e482afeb9191500" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41577 } } peers { permanent_uuid: "12899fec5f5c4194af251b18585e0469" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 38119 } }
I20260924 16:15:37.494777 12800 raft_consensus.cc:374] T 88626003aab141af8bb168b8f0454bb7 P 12899fec5f5c4194af251b18585e0469 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:15:37.494913 12800 raft_consensus.cc:729] T 88626003aab141af8bb168b8f0454bb7 P 12899fec5f5c4194af251b18585e0469 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12899fec5f5c4194af251b18585e0469, State: Initialized, Role: FOLLOWER
I20260924 16:15:37.495639 12800 consensus_queue.cc:260] T 88626003aab141af8bb168b8f0454bb7 P 12899fec5f5c4194af251b18585e0469 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f991653d74f04af180776405e08e2bd0" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 42789 } } peers { permanent_uuid: "493291c4622e44f19e482afeb9191500" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41577 } } peers { permanent_uuid: "12899fec5f5c4194af251b18585e0469" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 38119 } }
I20260924 16:15:37.496089 12799 ts_tablet_manager.cc:1428] T 88626003aab141af8bb168b8f0454bb7 P f991653d74f04af180776405e08e2bd0: Time spent starting tablet: real 0.010s user 0.005s sys 0.005s
I20260924 16:15:37.495922 12798 raft_consensus.cc:348] T 88626003aab141af8bb168b8f0454bb7 P 493291c4622e44f19e482afeb9191500 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f991653d74f04af180776405e08e2bd0" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 42789 } } peers { permanent_uuid: "493291c4622e44f19e482afeb9191500" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41577 } } peers { permanent_uuid: "12899fec5f5c4194af251b18585e0469" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 38119 } }
I20260924 16:15:37.496511 12798 raft_consensus.cc:374] T 88626003aab141af8bb168b8f0454bb7 P 493291c4622e44f19e482afeb9191500 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:15:37.496693 12798 raft_consensus.cc:729] T 88626003aab141af8bb168b8f0454bb7 P 493291c4622e44f19e482afeb9191500 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 493291c4622e44f19e482afeb9191500, State: Initialized, Role: FOLLOWER
I20260924 16:15:37.497159 12780 heartbeater.cc:503] Master 127.29.234.62:35061 was elected leader, sending a full tablet report...
I20260924 16:15:37.497418 12798 consensus_queue.cc:260] T 88626003aab141af8bb168b8f0454bb7 P 493291c4622e44f19e482afeb9191500 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f991653d74f04af180776405e08e2bd0" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 42789 } } peers { permanent_uuid: "493291c4622e44f19e482afeb9191500" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41577 } } peers { permanent_uuid: "12899fec5f5c4194af251b18585e0469" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 38119 } }
I20260924 16:15:37.497949 12800 ts_tablet_manager.cc:1428] T 88626003aab141af8bb168b8f0454bb7 P 12899fec5f5c4194af251b18585e0469: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260924 16:15:37.499334 12645 heartbeater.cc:503] Master 127.29.234.62:35061 was elected leader, sending a full tablet report...
I20260924 16:15:37.499867 12798 ts_tablet_manager.cc:1428] T 88626003aab141af8bb168b8f0454bb7 P 493291c4622e44f19e482afeb9191500: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
W20260924 16:15:37.592777 12509 tablet.cc:2369] T 88626003aab141af8bb168b8f0454bb7 P f991653d74f04af180776405e08e2bd0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 16:15:37.600263 12646 tablet.cc:2369] T 88626003aab141af8bb168b8f0454bb7 P 493291c4622e44f19e482afeb9191500: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 16:15:37.614135 12781 tablet.cc:2369] T 88626003aab141af8bb168b8f0454bb7 P 12899fec5f5c4194af251b18585e0469: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:15:37.697995 12806 raft_consensus.cc:482] T 88626003aab141af8bb168b8f0454bb7 P 493291c4622e44f19e482afeb9191500 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:15:37.698348 12806 raft_consensus.cc:504] T 88626003aab141af8bb168b8f0454bb7 P 493291c4622e44f19e482afeb9191500 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f991653d74f04af180776405e08e2bd0" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 42789 } } peers { permanent_uuid: "493291c4622e44f19e482afeb9191500" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41577 } } peers { permanent_uuid: "12899fec5f5c4194af251b18585e0469" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 38119 } }
I20260924 16:15:37.700254 12806 leader_election.cc:288] T 88626003aab141af8bb168b8f0454bb7 P 493291c4622e44f19e482afeb9191500 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f991653d74f04af180776405e08e2bd0 (127.29.234.1:42789), 12899fec5f5c4194af251b18585e0469 (127.29.234.3:38119)
I20260924 16:15:37.708606 12797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:37.700741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.2:48045 (local address 127.29.234.3:38119)
0924 16:15:37.700950 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:37.700957 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:37.701858 (+ 901us) server_negotiation.cc:407] Connection header received
0924 16:15:37.702100 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:37.702106 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:37.702256 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:37.702363 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:15:37.702871 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:37.703806 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:37.706225 (+ 2419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:37.707472 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:37.707621 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:37.707691 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:37.708223 (+ 532us) server_negotiation.cc:299] Negotiation successful
0924 16:15:37.708348 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 16:15:37.708715 12807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:37.700731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.3:38119 (local address 127.29.234.2:48045)
0924 16:15:37.701726 (+ 995us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:37.701748 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:37.701904 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:37.702378 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:37.702383 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:37.702410 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:15:37.702717 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:37.702728 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:37.703933 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:37.703940 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:37.706041 (+ 2101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:37.706057 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:37.707690 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:37.707698 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:37.707923 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:37.708063 (+ 140us) client_negotiation.cc:769] Sending connection context
0924 16:15:37.708183 (+ 120us) client_negotiation.cc:240] Negotiation successful
0924 16:15:37.708320 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":867,"mutex_wait_us":70,"thread_start_us":633,"threads_started":1}
I20260924 16:15:37.708784 12793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:37.700800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.2:53407 (local address 127.29.234.1:42789)
0924 16:15:37.700997 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:37.701002 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:37.702291 (+ 1289us) server_negotiation.cc:407] Connection header received
0924 16:15:37.702497 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:37.702504 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:37.702581 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:37.702637 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:15:37.703401 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:37.704469 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:37.706732 (+ 2263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:37.707782 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:37.707949 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:37.708023 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:37.708404 (+ 381us) server_negotiation.cc:299] Negotiation successful
0924 16:15:37.708546 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 16:15:37.709522 12808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:37.700731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:42789 (local address 127.29.234.2:53407)
0924 16:15:37.702162 (+ 1431us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:37.702183 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:37.702313 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:37.702808 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:37.702814 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:37.702840 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:15:37.703235 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:37.703248 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:37.704624 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:37.704630 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:37.706573 (+ 1943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:37.706585 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:37.707995 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:37.708001 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:37.708224 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:37.708258 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 16:15:37.709015 (+ 757us) client_negotiation.cc:240] Negotiation successful
0924 16:15:37.709152 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1307,"mutex_wait_us":33,"thread_start_us":476,"threads_started":1}
I20260924 16:15:37.709998 12733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88626003aab141af8bb168b8f0454bb7" candidate_uuid: "493291c4622e44f19e482afeb9191500" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12899fec5f5c4194af251b18585e0469" is_pre_election: true
I20260924 16:15:37.710495 12733 raft_consensus.cc:2454] T 88626003aab141af8bb168b8f0454bb7 P 12899fec5f5c4194af251b18585e0469 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 493291c4622e44f19e482afeb9191500 in term 0.
I20260924 16:15:37.711213 12531 leader_election.cc:302] T 88626003aab141af8bb168b8f0454bb7 P 493291c4622e44f19e482afeb9191500 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12899fec5f5c4194af251b18585e0469, 493291c4622e44f19e482afeb9191500; no voters:
I20260924 16:15:37.711642 12806 raft_consensus.cc:2781] T 88626003aab141af8bb168b8f0454bb7 P 493291c4622e44f19e482afeb9191500 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:15:37.711787 12806 raft_consensus.cc:482] T 88626003aab141af8bb168b8f0454bb7 P 493291c4622e44f19e482afeb9191500 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:15:37.711876 12806 raft_consensus.cc:3037] T 88626003aab141af8bb168b8f0454bb7 P 493291c4622e44f19e482afeb9191500 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:15:37.711848 12461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88626003aab141af8bb168b8f0454bb7" candidate_uuid: "493291c4622e44f19e482afeb9191500" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f991653d74f04af180776405e08e2bd0" is_pre_election: true
I20260924 16:15:37.712368 12461 raft_consensus.cc:2454] T 88626003aab141af8bb168b8f0454bb7 P f991653d74f04af180776405e08e2bd0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 493291c4622e44f19e482afeb9191500 in term 0.
I20260924 16:15:37.714998 12806 raft_consensus.cc:504] T 88626003aab141af8bb168b8f0454bb7 P 493291c4622e44f19e482afeb9191500 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f991653d74f04af180776405e08e2bd0" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 42789 } } peers { permanent_uuid: "493291c4622e44f19e482afeb9191500" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41577 } } peers { permanent_uuid: "12899fec5f5c4194af251b18585e0469" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 38119 } }
I20260924 16:15:37.715667 12806 leader_election.cc:288] T 88626003aab141af8bb168b8f0454bb7 P 493291c4622e44f19e482afeb9191500 [CANDIDATE]: Term 1 election: Requested vote from peers f991653d74f04af180776405e08e2bd0 (127.29.234.1:42789), 12899fec5f5c4194af251b18585e0469 (127.29.234.3:38119)
I20260924 16:15:37.716084 12733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88626003aab141af8bb168b8f0454bb7" candidate_uuid: "493291c4622e44f19e482afeb9191500" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12899fec5f5c4194af251b18585e0469"
I20260924 16:15:37.716090 12461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88626003aab141af8bb168b8f0454bb7" candidate_uuid: "493291c4622e44f19e482afeb9191500" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f991653d74f04af180776405e08e2bd0"
I20260924 16:15:37.716275 12461 raft_consensus.cc:3037] T 88626003aab141af8bb168b8f0454bb7 P f991653d74f04af180776405e08e2bd0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:15:37.716292 12733 raft_consensus.cc:3037] T 88626003aab141af8bb168b8f0454bb7 P 12899fec5f5c4194af251b18585e0469 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:15:37.718529 12733 raft_consensus.cc:2454] T 88626003aab141af8bb168b8f0454bb7 P 12899fec5f5c4194af251b18585e0469 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 493291c4622e44f19e482afeb9191500 in term 1.
I20260924 16:15:37.718959 12531 leader_election.cc:302] T 88626003aab141af8bb168b8f0454bb7 P 493291c4622e44f19e482afeb9191500 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12899fec5f5c4194af251b18585e0469, 493291c4622e44f19e482afeb9191500; no voters:
I20260924 16:15:37.719237 12806 raft_consensus.cc:2781] T 88626003aab141af8bb168b8f0454bb7 P 493291c4622e44f19e482afeb9191500 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:15:37.719388 12461 raft_consensus.cc:2454] T 88626003aab141af8bb168b8f0454bb7 P f991653d74f04af180776405e08e2bd0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 493291c4622e44f19e482afeb9191500 in term 1.
I20260924 16:15:37.719800 12806 raft_consensus.cc:686] T 88626003aab141af8bb168b8f0454bb7 P 493291c4622e44f19e482afeb9191500 [term 1 LEADER]: Becoming Leader. State: Replica: 493291c4622e44f19e482afeb9191500, State: Running, Role: LEADER
I20260924 16:15:37.720332 12806 consensus_queue.cc:237] T 88626003aab141af8bb168b8f0454bb7 P 493291c4622e44f19e482afeb9191500 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f991653d74f04af180776405e08e2bd0" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 42789 } } peers { permanent_uuid: "493291c4622e44f19e482afeb9191500" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41577 } } peers { permanent_uuid: "12899fec5f5c4194af251b18585e0469" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 38119 } }
I20260924 16:15:37.725338 12313 catalog_manager.cc:5495] T 88626003aab141af8bb168b8f0454bb7 P 493291c4622e44f19e482afeb9191500 reported cstate change: term changed from 0 to 1, leader changed from <none> to 493291c4622e44f19e482afeb9191500 (127.29.234.2). New cstate: current_term: 1 leader_uuid: "493291c4622e44f19e482afeb9191500" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f991653d74f04af180776405e08e2bd0" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 42789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "493291c4622e44f19e482afeb9191500" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12899fec5f5c4194af251b18585e0469" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 38119 } health_report { overall_health: UNKNOWN } } }
I20260924 16:15:37.806744 12795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:37.800707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46860 (local address 127.29.234.2:41577)
0924 16:15:37.800919 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:37.800926 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:37.800970 (+ 44us) server_negotiation.cc:407] Connection header received
0924 16:15:37.801242 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:37.801246 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:37.801323 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:37.801498 (+ 175us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:15:37.801902 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:37.802675 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:37.804141 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:37.804704 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:37.804842 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:37.805557 (+ 715us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:15:37.806135 (+ 578us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:15:37.806324 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:37.806376 (+ 52us) server_negotiation.cc:299] Negotiation successful
0924 16:15:37.806459 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 16:15:37.815201 12461 raft_consensus.cc:1261] T 88626003aab141af8bb168b8f0454bb7 P f991653d74f04af180776405e08e2bd0 [term 1 FOLLOWER]: Refusing update from remote peer 493291c4622e44f19e482afeb9191500: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:15:37.815272 12733 raft_consensus.cc:1261] T 88626003aab141af8bb168b8f0454bb7 P 12899fec5f5c4194af251b18585e0469 [term 1 FOLLOWER]: Refusing update from remote peer 493291c4622e44f19e482afeb9191500: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:15:37.816218 12809 consensus_queue.cc:1035] T 88626003aab141af8bb168b8f0454bb7 P 493291c4622e44f19e482afeb9191500 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12899fec5f5c4194af251b18585e0469" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:15:37.816627 12806 consensus_queue.cc:1035] T 88626003aab141af8bb168b8f0454bb7 P 493291c4622e44f19e482afeb9191500 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f991653d74f04af180776405e08e2bd0" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 42789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:15:37.845640 12314 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36734:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:15:37.845952 12314 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36734:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:15:37.850023 12314 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c32f836d405347ed9b2b10575a31a604: Sending DeleteTablet for 3 replicas of tablet 88626003aab141af8bb168b8f0454bb7
I20260924 16:15:37.851208 12440 tablet_service.cc:1514] Processing DeleteTablet for tablet 88626003aab141af8bb168b8f0454bb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:15:37 UTC) from {username='kudu', principal='kudu/127.29.234.62@KRBTEST.COM'} at 127.0.0.1:52964
I20260924 16:15:37.851562 12578 tablet_service.cc:1514] Processing DeleteTablet for tablet 88626003aab141af8bb168b8f0454bb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:15:37 UTC) from {username='kudu', principal='kudu/127.29.234.62@KRBTEST.COM'} at 127.0.0.1:46858
I20260924 16:15:37.851603 12713 tablet_service.cc:1514] Processing DeleteTablet for tablet 88626003aab141af8bb168b8f0454bb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:15:37 UTC) from {username='kudu', principal='kudu/127.29.234.62@KRBTEST.COM'} at 127.0.0.1:51568
I20260924 16:15:37.851964 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 12377
I20260924 16:15:37.871884 12823 tablet_replica.cc:330] T 88626003aab141af8bb168b8f0454bb7 P 493291c4622e44f19e482afeb9191500: stopping tablet replica
I20260924 16:15:37.872498 12823 raft_consensus.cc:2229] T 88626003aab141af8bb168b8f0454bb7 P 493291c4622e44f19e482afeb9191500 [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:15:37.873442 12823 raft_consensus.cc:2258] T 88626003aab141af8bb168b8f0454bb7 P 493291c4622e44f19e482afeb9191500 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:15:37.879391 12822 tablet_replica.cc:330] T 88626003aab141af8bb168b8f0454bb7 P f991653d74f04af180776405e08e2bd0: stopping tablet replica
I20260924 16:15:37.880151 12822 raft_consensus.cc:2229] T 88626003aab141af8bb168b8f0454bb7 P f991653d74f04af180776405e08e2bd0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:15:37.882464 12823 ts_tablet_manager.cc:1905] T 88626003aab141af8bb168b8f0454bb7 P 493291c4622e44f19e482afeb9191500: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:15:37.885684 12824 tablet_replica.cc:330] T 88626003aab141af8bb168b8f0454bb7 P 12899fec5f5c4194af251b18585e0469: stopping tablet replica
I20260924 16:15:37.886086 12824 raft_consensus.cc:2229] T 88626003aab141af8bb168b8f0454bb7 P 12899fec5f5c4194af251b18585e0469 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:15:37.886579 12824 raft_consensus.cc:2258] T 88626003aab141af8bb168b8f0454bb7 P 12899fec5f5c4194af251b18585e0469 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:15:37.887142 12822 raft_consensus.cc:2258] T 88626003aab141af8bb168b8f0454bb7 P f991653d74f04af180776405e08e2bd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:15:37.888995 12822 ts_tablet_manager.cc:1905] T 88626003aab141af8bb168b8f0454bb7 P f991653d74f04af180776405e08e2bd0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:15:37.890368 12823 ts_tablet_manager.cc:1918] T 88626003aab141af8bb168b8f0454bb7 P 493291c4622e44f19e482afeb9191500: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:15:37.890628 12823 log.cc:1196] T 88626003aab141af8bb168b8f0454bb7 P 493291c4622e44f19e482afeb9191500: Deleting WAL directory at /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-1/wal/wals/88626003aab141af8bb168b8f0454bb7
I20260924 16:15:37.891340 12823 ts_tablet_manager.cc:1939] T 88626003aab141af8bb168b8f0454bb7 P 493291c4622e44f19e482afeb9191500: Deleting consensus metadata
I20260924 16:15:37.893292 12300 catalog_manager.cc:4841] TS 493291c4622e44f19e482afeb9191500 (127.29.234.2:41577): tablet 88626003aab141af8bb168b8f0454bb7 (table test-table [id=407fe6ae8f3241b496968694d5bbe06a]) successfully deleted
Sep 24 16:15:37 dist-test-slave-v5j9 krb5kdc[12269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.234.1@KRBTEST.COM
I20260924 16:15:37.891083 12824 ts_tablet_manager.cc:1905] T 88626003aab141af8bb168b8f0454bb7 P 12899fec5f5c4194af251b18585e0469: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:15:37.898319 12822 ts_tablet_manager.cc:1918] T 88626003aab141af8bb168b8f0454bb7 P f991653d74f04af180776405e08e2bd0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:15:37.898928 12822 log.cc:1196] T 88626003aab141af8bb168b8f0454bb7 P f991653d74f04af180776405e08e2bd0: Deleting WAL directory at /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-0/wal/wals/88626003aab141af8bb168b8f0454bb7
I20260924 16:15:37.899655 12822 ts_tablet_manager.cc:1939] T 88626003aab141af8bb168b8f0454bb7 P f991653d74f04af180776405e08e2bd0: Deleting consensus metadata
I20260924 16:15:37.901703 12299 catalog_manager.cc:4841] TS f991653d74f04af180776405e08e2bd0 (127.29.234.1:42789): tablet 88626003aab141af8bb168b8f0454bb7 (table test-table [id=407fe6ae8f3241b496968694d5bbe06a]) successfully deleted
I20260924 16:15:37.903266 12824 ts_tablet_manager.cc:1918] T 88626003aab141af8bb168b8f0454bb7 P 12899fec5f5c4194af251b18585e0469: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:15:37.903458 12824 log.cc:1196] T 88626003aab141af8bb168b8f0454bb7 P 12899fec5f5c4194af251b18585e0469: Deleting WAL directory at /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-2/wal/wals/88626003aab141af8bb168b8f0454bb7
I20260924 16:15:37.903910 12793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:37.853506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52968 (local address 127.29.234.1:42789)
0924 16:15:37.853741 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:37.853751 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:37.875366 (+ 21615us) server_negotiation.cc:407] Connection header received
0924 16:15:37.875998 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:37.876007 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:37.876146 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:37.876225 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:37.880903 (+ 4678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:37.882463 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:37.889553 (+ 7090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:37.890241 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:37.890351 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:37.896342 (+ 5991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:37.896357 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:37.896376 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:15:37.896423 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:37.900461 (+ 4038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:37.901421 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:37.901430 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:37.901433 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:37.901603 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:37.902047 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:37.902051 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:37.902054 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:37.902400 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:37.902597 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:37.903163 (+ 566us) server_negotiation.cc:299] Negotiation successful
0924 16:15:37.903408 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 16:15:37.903988 12824 ts_tablet_manager.cc:1939] T 88626003aab141af8bb168b8f0454bb7 P 12899fec5f5c4194af251b18585e0469: Deleting consensus metadata
I20260924 16:15:37.904556 12503 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:15:37.905712 12300 catalog_manager.cc:4841] TS 12899fec5f5c4194af251b18585e0469 (127.29.234.3:38119): tablet 88626003aab141af8bb168b8f0454bb7 (table test-table [id=407fe6ae8f3241b496968694d5bbe06a]) successfully deleted
I20260924 16:15:37.996821 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 12377
I20260924 16:15:38.016322 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 12514
Sep 24 16:15:38 dist-test-slave-v5j9 krb5kdc[12269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.234.2@KRBTEST.COM
I20260924 16:15:38.038362 12795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:38.017215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46862 (local address 127.29.234.2:41577)
0924 16:15:38.017432 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:38.017439 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:38.017462 (+ 23us) server_negotiation.cc:407] Connection header received
0924 16:15:38.017533 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:38.017538 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:38.017618 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:38.017732 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:38.018968 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:38.020258 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:38.021595 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:38.022301 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:38.022435 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:38.031152 (+ 8717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:38.031172 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:38.031205 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:15:38.031259 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:38.035289 (+ 4030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:38.036007 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:38.036017 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:38.036021 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:38.036193 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:38.036549 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:38.036556 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:38.036564 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:38.037066 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:38.037450 (+ 384us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:38.037687 (+ 237us) server_negotiation.cc:299] Negotiation successful
0924 16:15:38.037916 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 16:15:38.039216 12640 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:15:38.154462 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 12514
I20260924 16:15:38.174118 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 12650
Sep 24 16:15:38 dist-test-slave-v5j9 krb5kdc[12269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.234.3@KRBTEST.COM
I20260924 16:15:38.188756 12797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:38.174871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51570 (local address 127.29.234.3:38119)
0924 16:15:38.175052 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:38.175060 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:38.175195 (+ 135us) server_negotiation.cc:407] Connection header received
0924 16:15:38.175411 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:38.175424 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:38.175490 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:38.175595 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:38.176795 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:38.177570 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:38.178796 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:38.179181 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:38.179315 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:38.183566 (+ 4251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:38.183577 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:38.183594 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:15:38.183625 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:38.186392 (+ 2767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:38.187043 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:38.187050 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:38.187052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:38.187224 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:38.187536 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:38.187539 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:38.187541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:38.187892 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:38.188118 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:38.188344 (+ 226us) server_negotiation.cc:299] Negotiation successful
0924 16:15:38.188497 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 16:15:38.189503 12775 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:15:38.277968 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 12650
I20260924 16:15:38.295406 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 12282
I20260924 16:15:38.296797 12345 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:15:38.409969 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 12282
I20260924 16:15:38.431878 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:35061
--webserver_interface=127.29.234.62
--webserver_port=46067
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab
--principal=kudu/127.29.234.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:45085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.234.62:35061
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:15:38.759361 12830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:15:38.759665 12830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:15:38.759852 12830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:15:38.767244 12830 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:15:38.767364 12830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:15:38.767472 12830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:15:38.767586 12830 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:15:38.767674 12830 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:15:38.777715 12830 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:45085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.29.234.62:35061
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab
--principal=kudu/127.29.234.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:35061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.234.62
--webserver_port=46067
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:15:38.779110 12830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:15:38.780498 12830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:15:38.804303 12839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:38.804355 12837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:38.804265 12836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:15:38.806908 12830 server_base.cc:1034] running on GCE node
I20260924 16:15:38.807824 12830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:15:38.809525 12830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:15:38.810751 12830 hybrid_clock.cc:648] HybridClock initialized: now 1790266538810688 us; error 38 us; skew 500 ppm
I20260924 16:15:38.811215 12830 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:15:38 dist-test-slave-v5j9 krb5kdc[12269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266538, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:15:38.821435 12830 init.cc:377] Logged in from keytab as kudu/127.29.234.62@KRBTEST.COM (short username kudu)
I20260924 16:15:38.823377 12830 webserver.cc:460] Webserver started at http://127.29.234.62:46067/ using document root <none> and password file <none>
I20260924 16:15:38.824270 12830 fs_manager.cc:359] Metadata directory not provided
I20260924 16:15:38.824453 12830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:15:38.831228 12830 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 16:15:38.835084 12846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:38.836760 12830 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:15:38.836975 12830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/master-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/master-0/wal
uuid: "c32f836d405347ed9b2b10575a31a604"
format_stamp: "Formatted at 2026-09-24 16:15:35 on dist-test-slave-v5j9"
server_key: "534e502a9197cb96561efb4001391942"
server_key_iv: "3fe3b87a490785895b4bbba7ead71661"
server_key_version: "encryptionkey@0"
I20260924 16:15:38.838294 12830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:15:38.859495 12830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:15:38.863870 12830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:15:38.864177 12830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:15:38.883837 12830 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.62:35061
I20260924 16:15:38.883867 12898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.62:35061 every 8 connection(s)
I20260924 16:15:38.885524 12830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
I20260924 16:15:38.893234 12899 sys_catalog.cc:263] Verifying existing consensus state
I20260924 16:15:38.893874 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 12830
I20260924 16:15:38.894882 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.1:42789
--local_ip_for_outbound_sockets=127.29.234.1
--tserver_master_addrs=127.29.234.62:35061
--webserver_port=40655
--webserver_interface=127.29.234.1
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab
--principal=kudu/127.29.234.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:45085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:15:38.898653 12899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c32f836d405347ed9b2b10575a31a604: Bootstrap starting.
I20260924 16:15:38.920835 12899 log.cc:824] T 00000000000000000000000000000000 P c32f836d405347ed9b2b10575a31a604: Log is configured to *not* fsync() on all Append() calls
I20260924 16:15:38.933131 12899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c32f836d405347ed9b2b10575a31a604: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 16:15:38.933823 12899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c32f836d405347ed9b2b10575a31a604: Bootstrap complete.
I20260924 16:15:38.943179 12899 raft_consensus.cc:348] T 00000000000000000000000000000000 P c32f836d405347ed9b2b10575a31a604 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c32f836d405347ed9b2b10575a31a604" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 35061 } }
I20260924 16:15:38.944087 12899 raft_consensus.cc:729] T 00000000000000000000000000000000 P c32f836d405347ed9b2b10575a31a604 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c32f836d405347ed9b2b10575a31a604, State: Initialized, Role: FOLLOWER
I20260924 16:15:38.944702 12899 consensus_queue.cc:260] T 00000000000000000000000000000000 P c32f836d405347ed9b2b10575a31a604 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c32f836d405347ed9b2b10575a31a604" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 35061 } }
I20260924 16:15:38.944934 12899 raft_consensus.cc:388] T 00000000000000000000000000000000 P c32f836d405347ed9b2b10575a31a604 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:15:38.945077 12899 raft_consensus.cc:482] T 00000000000000000000000000000000 P c32f836d405347ed9b2b10575a31a604 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:15:38.945253 12899 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c32f836d405347ed9b2b10575a31a604 [term 1 FOLLOWER]: Advancing to term 2
I20260924 16:15:38.949108 12899 raft_consensus.cc:504] T 00000000000000000000000000000000 P c32f836d405347ed9b2b10575a31a604 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c32f836d405347ed9b2b10575a31a604" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 35061 } }
I20260924 16:15:38.949630 12899 leader_election.cc:302] T 00000000000000000000000000000000 P c32f836d405347ed9b2b10575a31a604 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c32f836d405347ed9b2b10575a31a604; no voters:
I20260924 16:15:38.950260 12899 leader_election.cc:288] T 00000000000000000000000000000000 P c32f836d405347ed9b2b10575a31a604 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 16:15:38.950400 12903 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c32f836d405347ed9b2b10575a31a604 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 16:15:38.952726 12899 sys_catalog.cc:564] T 00000000000000000000000000000000 P c32f836d405347ed9b2b10575a31a604 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:15:38.952831 12903 raft_consensus.cc:686] T 00000000000000000000000000000000 P c32f836d405347ed9b2b10575a31a604 [term 2 LEADER]: Becoming Leader. State: Replica: c32f836d405347ed9b2b10575a31a604, State: Running, Role: LEADER
I20260924 16:15:38.953534 12903 consensus_queue.cc:237] T 00000000000000000000000000000000 P c32f836d405347ed9b2b10575a31a604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c32f836d405347ed9b2b10575a31a604" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 35061 } }
I20260924 16:15:38.962494 12905 sys_catalog.cc:455] T 00000000000000000000000000000000 P c32f836d405347ed9b2b10575a31a604 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c32f836d405347ed9b2b10575a31a604. Latest consensus state: current_term: 2 leader_uuid: "c32f836d405347ed9b2b10575a31a604" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c32f836d405347ed9b2b10575a31a604" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 35061 } } }
I20260924 16:15:38.962903 12905 sys_catalog.cc:458] T 00000000000000000000000000000000 P c32f836d405347ed9b2b10575a31a604 [sys.catalog]: This master's current role is: LEADER
I20260924 16:15:38.963411 12904 sys_catalog.cc:455] T 00000000000000000000000000000000 P c32f836d405347ed9b2b10575a31a604 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c32f836d405347ed9b2b10575a31a604" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c32f836d405347ed9b2b10575a31a604" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 35061 } } }
I20260924 16:15:38.963675 12904 sys_catalog.cc:458] T 00000000000000000000000000000000 P c32f836d405347ed9b2b10575a31a604 [sys.catalog]: This master's current role is: LEADER
I20260924 16:15:38.965108 12919 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:15:38.975454 12919 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:15:38.976492 12919 catalog_manager.cc:1259] Loaded cluster ID: 5f4cda6912f7450594aa256ddbe1ceb6
I20260924 16:15:38.976601 12919 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:15:38.978943 12919 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260924 16:15:38.989696 12919 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260924 16:15:38.997234 12919 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:15:38.998775 12919 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c32f836d405347ed9b2b10575a31a604: Loaded TSK: 0
I20260924 16:15:38.999899 12919 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 16:15:39.266242 12901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:15:39.266573 12901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:15:39.266739 12901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:15:39.274164 12901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:15:39.274317 12901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:15:39.274485 12901 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.1
I20260924 16:15:39.284199 12901 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:45085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab
--principal=kudu/127.29.234.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.1:42789
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.234.1
--webserver_port=40655
--tserver_master_addrs=127.29.234.62:35061
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.1
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:15:39.285622 12901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:15:39.286988 12901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:15:39.302294 12929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:39.302338 12928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:39.303438 12931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:15:39.306919 12901 server_base.cc:1034] running on GCE node
I20260924 16:15:39.307628 12901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:15:39.308607 12901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:15:39.309847 12901 hybrid_clock.cc:648] HybridClock initialized: now 1790266539309784 us; error 54 us; skew 500 ppm
I20260924 16:15:39.310182 12901 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:15:39 dist-test-slave-v5j9 krb5kdc[12269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266539, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:15:39.313436 12901 init.cc:377] Logged in from keytab as kudu/127.29.234.1@KRBTEST.COM (short username kudu)
I20260924 16:15:39.314993 12901 webserver.cc:460] Webserver started at http://127.29.234.1:40655/ using document root <none> and password file <none>
I20260924 16:15:39.315706 12901 fs_manager.cc:359] Metadata directory not provided
I20260924 16:15:39.315845 12901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:15:39.321796 12901 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 16:15:39.324698 12938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:39.326445 12901 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:15:39.326658 12901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-0/wal
uuid: "f991653d74f04af180776405e08e2bd0"
format_stamp: "Formatted at 2026-09-24 16:15:36 on dist-test-slave-v5j9"
server_key: "c8b280f8598c43a400bc4fdeb8e2648f"
server_key_iv: "0d2a977e2b1d6e86b92f43b018962069"
server_key_version: "encryptionkey@0"
I20260924 16:15:39.327729 12901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:15:39.355237 12901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:15:39.359797 12901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:15:39.360098 12901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:15:39.361160 12901 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:15:39.362605 12901 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:15:39.362740 12901 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:39.362883 12901 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:15:39.362974 12901 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:39.395655 12901 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.1:42789
I20260924 16:15:39.395743 13051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.1:42789 every 8 connection(s)
I20260924 16:15:39.397150 12901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
I20260924 16:15:39.402880 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 12901
I20260924 16:15:39.403728 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.2:41577
--local_ip_for_outbound_sockets=127.29.234.2
--tserver_master_addrs=127.29.234.62:35061
--webserver_port=44075
--webserver_interface=127.29.234.2
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab
--principal=kudu/127.29.234.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:45085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 16:15:39 dist-test-slave-v5j9 krb5kdc[12269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266539, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.1@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:15:39.423009 13055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:39.400233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:35061 (local address 127.29.234.1:46107)
0924 16:15:39.401038 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:39.401082 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:39.402929 (+ 1847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:39.404718 (+ 1789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:39.404726 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:39.405260 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:15:39.405888 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:39.405907 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:39.408356 (+ 2449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:39.408360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:39.409702 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:39.409715 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:39.411232 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:39.411242 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:39.411560 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:39.412353 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:15:39.412388 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:15:39.415061 (+ 2673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:15:39.418688 (+ 3627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:15:39.418697 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:15:39.418715 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:15:39.419153 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:15:39.419620 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:15:39.419624 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:15:39.419644 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:15:39.419835 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:15:39.420603 (+ 768us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:15:39.420611 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:15:39.420979 (+ 368us) client_negotiation.cc:769] Sending connection context
0924 16:15:39.421267 (+ 288us) client_negotiation.cc:240] Negotiation successful
0924 16:15:39.421792 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":343,"threads_started":1}
I20260924 16:15:39.423548 13054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:39.400112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.1:46107 (local address 127.29.234.62:35061)
0924 16:15:39.400716 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:39.400728 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:39.401228 (+ 500us) server_negotiation.cc:407] Connection header received
0924 16:15:39.403618 (+ 2390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:39.403641 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:39.404176 (+ 535us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:39.404597 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:39.406157 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:39.408186 (+ 2029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:39.409847 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:39.410949 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:39.411184 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:39.415370 (+ 4186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:39.415378 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:39.415394 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:15:39.415430 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:39.418497 (+ 3067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:39.419313 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:39.419320 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:39.419325 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:39.419483 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:39.419958 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:39.419967 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:39.419970 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:39.420458 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:39.420677 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:39.421615 (+ 938us) server_negotiation.cc:299] Negotiation successful
0924 16:15:39.422128 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":266,"threads_started":1}
I20260924 16:15:39.426404 13052 heartbeater.cc:347] Connected to a master server at 127.29.234.62:35061
I20260924 16:15:39.426743 13052 heartbeater.cc:464] Registering TS with master...
I20260924 16:15:39.427486 13052 heartbeater.cc:511] Master 127.29.234.62:35061 requested a full tablet report, sending...
I20260924 16:15:39.429765 12863 ts_manager.cc:194] Registered new tserver with Master: f991653d74f04af180776405e08e2bd0 (127.29.234.1:42789)
I20260924 16:15:39.431273 12863 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.234.1@KRBTEST.COM'} at 127.29.234.1:46107
W20260924 16:15:39.737109 13056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:15:39.737461 13056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:15:39.737635 13056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:15:39.745065 13056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:15:39.745227 13056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:15:39.745434 13056 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.2
I20260924 16:15:39.755610 13056 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:45085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab
--principal=kudu/127.29.234.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.2:41577
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.234.2
--webserver_port=44075
--tserver_master_addrs=127.29.234.62:35061
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.2
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:15:39.756982 13056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:15:39.758448 13056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:15:39.771896 13062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:39.771869 13061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:39.772101 13064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:15:39.772706 13056 server_base.cc:1034] running on GCE node
I20260924 16:15:39.773388 13056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:15:39.774338 13056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:15:39.775547 13056 hybrid_clock.cc:648] HybridClock initialized: now 1790266539775485 us; error 59 us; skew 500 ppm
I20260924 16:15:39.775897 13056 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:15:39 dist-test-slave-v5j9 krb5kdc[12269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266539, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:15:39.779251 13056 init.cc:377] Logged in from keytab as kudu/127.29.234.2@KRBTEST.COM (short username kudu)
I20260924 16:15:39.780754 13056 webserver.cc:460] Webserver started at http://127.29.234.2:44075/ using document root <none> and password file <none>
I20260924 16:15:39.781517 13056 fs_manager.cc:359] Metadata directory not provided
I20260924 16:15:39.781685 13056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:15:39.787019 13056 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:15:39.789743 13071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:39.791265 13056 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 16:15:39.791442 13056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-1/wal
uuid: "493291c4622e44f19e482afeb9191500"
format_stamp: "Formatted at 2026-09-24 16:15:36 on dist-test-slave-v5j9"
server_key: "277fe4ba9eb1034086a741e02efada08"
server_key_iv: "a93b1f0c7143d992a7e5f8fb6e984152"
server_key_version: "encryptionkey@0"
I20260924 16:15:39.792479 13056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:15:39.812500 13056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:15:39.816735 13056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:15:39.817003 13056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:15:39.817991 13056 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:15:39.819298 13056 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:15:39.819416 13056 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:39.819551 13056 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:15:39.819670 13056 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:39.855207 13056 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.2:41577
I20260924 16:15:39.855227 13186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.2:41577 every 8 connection(s)
I20260924 16:15:39.856899 13056 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
I20260924 16:15:39.859450 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 13056
I20260924 16:15:39.860194 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.3:38119
--local_ip_for_outbound_sockets=127.29.234.3
--tserver_master_addrs=127.29.234.62:35061
--webserver_port=44797
--webserver_interface=127.29.234.3
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab
--principal=kudu/127.29.234.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:45085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 16:15:39 dist-test-slave-v5j9 krb5kdc[12269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266539, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.2@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:15:39.877521 13054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:39.859656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.2:33291 (local address 127.29.234.62:35061)
0924 16:15:39.859843 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:39.859848 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:39.861320 (+ 1472us) server_negotiation.cc:407] Connection header received
0924 16:15:39.863198 (+ 1878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:39.863204 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:39.863306 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:39.863421 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:39.865007 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:39.865791 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:39.867430 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:39.867865 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:39.868000 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:39.871742 (+ 3742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:39.871748 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:39.871751 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:15:39.871785 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:39.874404 (+ 2619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:39.875134 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:39.875142 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:39.875146 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:39.875294 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:39.875731 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:39.875738 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:39.875741 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:39.876016 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:39.876193 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:39.876878 (+ 685us) server_negotiation.cc:299] Negotiation successful
0924 16:15:39.877113 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 16:15:39.878536 13189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:39.860383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:35061 (local address 127.29.234.2:33291)
0924 16:15:39.861149 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:39.861202 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:39.862894 (+ 1692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:39.863687 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:39.863697 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:39.864226 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:15:39.864829 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:39.864849 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:39.866003 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:39.866007 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:39.867274 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:39.867287 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:39.868111 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:39.868119 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:39.868437 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:39.869200 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:15:39.869229 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:15:39.871504 (+ 2275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:15:39.874559 (+ 3055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:15:39.874574 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:15:39.874588 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:15:39.875009 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:15:39.875410 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:15:39.875415 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:15:39.875420 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:15:39.875610 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:15:39.876148 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:15:39.876158 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:15:39.876514 (+ 356us) client_negotiation.cc:769] Sending connection context
0924 16:15:39.876792 (+ 278us) client_negotiation.cc:240] Negotiation successful
0924 16:15:39.877336 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":312,"threads_started":1}
I20260924 16:15:39.880120 13187 heartbeater.cc:347] Connected to a master server at 127.29.234.62:35061
I20260924 16:15:39.880445 13187 heartbeater.cc:464] Registering TS with master...
I20260924 16:15:39.881191 13187 heartbeater.cc:511] Master 127.29.234.62:35061 requested a full tablet report, sending...
I20260924 16:15:39.882802 12863 ts_manager.cc:194] Registered new tserver with Master: 493291c4622e44f19e482afeb9191500 (127.29.234.2:41577)
I20260924 16:15:39.884184 12863 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.234.2@KRBTEST.COM'} at 127.29.234.2:33291
W20260924 16:15:40.202126 13190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:15:40.202400 13190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:15:40.202549 13190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:15:40.210049 13190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:15:40.210157 13190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:15:40.210388 13190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.3
I20260924 16:15:40.220288 13190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:45085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab
--principal=kudu/127.29.234.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.3:38119
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.234.3
--webserver_port=44797
--tserver_master_addrs=127.29.234.62:35061
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.3
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:15:40.221731 13190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:15:40.223063 13190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:15:40.236557 13195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:40.236514 13198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:40.236634 13196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:15:40.237567 13190 server_base.cc:1034] running on GCE node
I20260924 16:15:40.238111 13190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:15:40.239012 13190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:15:40.240250 13190 hybrid_clock.cc:648] HybridClock initialized: now 1790266540240204 us; error 59 us; skew 500 ppm
I20260924 16:15:40.240592 13190 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:15:40 dist-test-slave-v5j9 krb5kdc[12269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266540, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:15:40.243963 13190 init.cc:377] Logged in from keytab as kudu/127.29.234.3@KRBTEST.COM (short username kudu)
I20260924 16:15:40.245530 13190 webserver.cc:460] Webserver started at http://127.29.234.3:44797/ using document root <none> and password file <none>
I20260924 16:15:40.246309 13190 fs_manager.cc:359] Metadata directory not provided
I20260924 16:15:40.246470 13190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:15:40.251890 13190 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.003s
I20260924 16:15:40.254822 13205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:40.256327 13190 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:15:40.256529 13190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-2/wal
uuid: "12899fec5f5c4194af251b18585e0469"
format_stamp: "Formatted at 2026-09-24 16:15:37 on dist-test-slave-v5j9"
server_key: "6754491885e551fa0c9efa254bbfbcd9"
server_key_iv: "d977910b03c2242e1b2b3f4495f600b1"
server_key_version: "encryptionkey@0"
I20260924 16:15:40.257630 13190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:15:40.279237 13190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:15:40.283829 13190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:15:40.284124 13190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:15:40.285197 13190 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:15:40.286610 13190 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:15:40.286702 13190 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:40.286796 13190 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:15:40.286851 13190 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:40.320926 13190 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.3:38119
I20260924 16:15:40.321031 13321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.3:38119 every 8 connection(s)
I20260924 16:15:40.322557 13190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
I20260924 16:15:40.325337 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 13190
Sep 24 16:15:40 dist-test-slave-v5j9 krb5kdc[12269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266540, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.3@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:15:40.343077 13325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:40.326386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36742 (local address 127.29.234.62:35061)
0924 16:15:40.328234 (+ 1848us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:40.328240 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:40.328258 (+ 18us) server_negotiation.cc:407] Connection header received
0924 16:15:40.328319 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:40.328324 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:40.328402 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:40.328518 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:40.330147 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:40.331249 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:40.332676 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:40.333299 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:40.333448 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:40.336230 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:40.336237 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:40.336242 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:15:40.336281 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:40.339508 (+ 3227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:40.340485 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:40.340491 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:40.340494 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:40.340648 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:40.341105 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:40.341109 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:40.341111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:40.341573 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:40.341862 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:40.342302 (+ 440us) server_negotiation.cc:299] Negotiation successful
0924 16:15:40.342591 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1731,"thread_start_us":1619,"threads_started":1}
I20260924 16:15:40.346156 13054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:40.325135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.3:33641 (local address 127.29.234.62:35061)
0924 16:15:40.325315 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:40.325319 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:40.326940 (+ 1621us) server_negotiation.cc:407] Connection header received
0924 16:15:40.329440 (+ 2500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:40.329445 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:40.329528 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:40.329632 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:40.331624 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:40.332442 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:40.334632 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:40.335058 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:40.335193 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:40.339282 (+ 4089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:40.339292 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:40.339301 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:15:40.339358 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:40.342852 (+ 3494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:40.343843 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:40.343848 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:40.343852 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:40.343979 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:40.344427 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:40.344432 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:40.344435 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:40.344798 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:40.344944 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:40.345747 (+ 803us) server_negotiation.cc:299] Negotiation successful
0924 16:15:40.345901 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 16:15:40.347254 13324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:40.325881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:35061 (local address 127.29.234.3:33641)
0924 16:15:40.326771 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:40.326820 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:40.329133 (+ 2313us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:40.329898 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:40.329908 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:40.330650 (+ 742us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:15:40.331468 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:40.331489 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:40.332603 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:40.332607 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:40.334476 (+ 1869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:40.334488 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:40.335319 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:40.335328 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:40.335544 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:40.336221 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:15:40.336258 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:15:40.339030 (+ 2772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:15:40.342975 (+ 3945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:15:40.343004 (+ 29us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:15:40.343026 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:15:40.343704 (+ 678us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:15:40.344092 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:15:40.344097 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:15:40.344100 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:15:40.344303 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:15:40.344930 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:15:40.344937 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:15:40.345277 (+ 340us) client_negotiation.cc:769] Sending connection context
0924 16:15:40.345644 (+ 367us) client_negotiation.cc:240] Negotiation successful
0924 16:15:40.346109 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":358,"threads_started":1}
I20260924 16:15:40.348642 13322 heartbeater.cc:347] Connected to a master server at 127.29.234.62:35061
I20260924 16:15:40.348973 13322 heartbeater.cc:464] Registering TS with master...
I20260924 16:15:40.349797 13322 heartbeater.cc:511] Master 127.29.234.62:35061 requested a full tablet report, sending...
I20260924 16:15:40.351145 12863 ts_manager.cc:194] Registered new tserver with Master: 12899fec5f5c4194af251b18585e0469 (127.29.234.3:38119)
I20260924 16:15:40.351962 12863 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.234.3@KRBTEST.COM'} at 127.29.234.3:33641
I20260924 16:15:40.357919 30632 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:15:40.365419 13054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:40.358928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36746 (local address 127.29.234.62:35061)
0924 16:15:40.359198 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:40.359205 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:40.359512 (+ 307us) server_negotiation.cc:407] Connection header received
0924 16:15:40.359765 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:40.359769 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:40.359901 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:40.360056 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:15:40.360498 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:40.361451 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:40.363214 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:40.363655 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:40.363797 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:40.364321 (+ 524us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:15:40.364794 (+ 473us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:15:40.364951 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:40.365029 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 16:15:40.365108 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260924 16:15:40.366420 12863 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:36746:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:15:40.374191 12863 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:15:40.397568 13332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:40.385642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:42789 (local address 127.0.0.1:52978)
0924 16:15:40.386499 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:40.386537 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:40.386721 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:40.387670 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:40.387678 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:40.387716 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:15:40.388206 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:40.388218 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:40.390420 (+ 2202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:40.390425 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:40.393453 (+ 3028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:40.393466 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:40.396285 (+ 2819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:40.396294 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:40.396573 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:40.396646 (+ 73us) client_negotiation.cc:769] Sending connection context
0924 16:15:40.396940 (+ 294us) client_negotiation.cc:240] Negotiation successful
0924 16:15:40.397172 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":349,"threads_started":1}
I20260924 16:15:40.398706 13333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:40.385848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52978 (local address 127.29.234.1:42789)
0924 16:15:40.386819 (+ 971us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:40.386829 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:40.386856 (+ 27us) server_negotiation.cc:407] Connection header received
0924 16:15:40.386957 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:40.386966 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:40.387240 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:40.388206 (+ 966us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:15:40.388479 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:40.390260 (+ 1781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:40.393626 (+ 3366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:40.395556 (+ 1930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:40.395784 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:40.396044 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:40.397957 (+ 1913us) server_negotiation.cc:299] Negotiation successful
0924 16:15:40.398170 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":812,"thread_start_us":886,"threads_started":1}
I20260924 16:15:40.403626 12986 tablet_service.cc:1467] Processing CreateTablet for tablet ce3789dc5b2e4b85aa79863bad2ebf34 (DEFAULT_TABLE table=test-table [id=01941355f0d8474fbe1b57e6318322fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:15:40.405221 12986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce3789dc5b2e4b85aa79863bad2ebf34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:15:40.409654 13334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:40.386522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.3:38119 (local address 127.0.0.1:51572)
0924 16:15:40.387912 (+ 1390us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:40.387928 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:40.388034 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:40.402883 (+ 14849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:40.402889 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:40.402915 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:15:40.403340 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:40.403352 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:40.405481 (+ 2129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:40.405488 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:40.407007 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:40.407020 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:40.408740 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:40.408748 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:40.408934 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:40.408968 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 16:15:40.409034 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 16:15:40.409174 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1245,"mutex_wait_us":26,"thread_start_us":1088,"threads_started":1}
I20260924 16:15:40.410101 13336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:40.388089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.2:41577 (local address 127.0.0.1:46876)
0924 16:15:40.399443 (+ 11354us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:40.399463 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:40.399579 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:40.401644 (+ 2065us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:40.401648 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:40.401673 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:15:40.403154 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:40.403166 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:40.405090 (+ 1924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:40.405096 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:40.407324 (+ 2228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:40.407339 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:40.409187 (+ 1848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:40.409195 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:40.409420 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:40.409455 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 16:15:40.409570 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 16:15:40.409710 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11214,"mutex_wait_us":1,"thread_start_us":15652,"threads_started":1}
I20260924 16:15:40.410334 13335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:40.387327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51572 (local address 127.29.234.3:38119)
0924 16:15:40.402302 (+ 14975us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:40.402311 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:40.402344 (+ 33us) server_negotiation.cc:407] Connection header received
0924 16:15:40.402434 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:40.402444 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:40.402667 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:40.403543 (+ 876us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:15:40.403837 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:40.405349 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:40.407167 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:40.408491 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:40.409479 (+ 988us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:40.409730 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:40.409812 (+ 82us) server_negotiation.cc:299] Negotiation successful
0924 16:15:40.409987 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14778,"thread_start_us":16564,"threads_started":1}
I20260924 16:15:40.410794 13337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:40.388653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46876 (local address 127.29.234.2:41577)
0924 16:15:40.401050 (+ 12397us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:40.401059 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:40.401088 (+ 29us) server_negotiation.cc:407] Connection header received
0924 16:15:40.401187 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:40.401193 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:40.401448 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:40.401643 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:15:40.403381 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:40.404941 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:40.407496 (+ 2555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:40.408958 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:40.409091 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:40.409281 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:40.410322 (+ 1041us) server_negotiation.cc:299] Negotiation successful
0924 16:15:40.410488 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12235,"spinlock_wait_cycles":21888,"thread_start_us":12211,"threads_started":1}
I20260924 16:15:40.413818 13119 tablet_service.cc:1467] Processing CreateTablet for tablet ce3789dc5b2e4b85aa79863bad2ebf34 (DEFAULT_TABLE table=test-table [id=01941355f0d8474fbe1b57e6318322fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:15:40.414072 13256 tablet_service.cc:1467] Processing CreateTablet for tablet ce3789dc5b2e4b85aa79863bad2ebf34 (DEFAULT_TABLE table=test-table [id=01941355f0d8474fbe1b57e6318322fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:15:40.415102 13119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce3789dc5b2e4b85aa79863bad2ebf34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:15:40.415441 13256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce3789dc5b2e4b85aa79863bad2ebf34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:15:40.421096 13338 tablet_bootstrap.cc:492] T ce3789dc5b2e4b85aa79863bad2ebf34 P f991653d74f04af180776405e08e2bd0: Bootstrap starting.
I20260924 16:15:40.424741 13338 tablet_bootstrap.cc:654] T ce3789dc5b2e4b85aa79863bad2ebf34 P f991653d74f04af180776405e08e2bd0: Neither blocks nor log segments found. Creating new log.
I20260924 16:15:40.426056 13338 log.cc:824] T ce3789dc5b2e4b85aa79863bad2ebf34 P f991653d74f04af180776405e08e2bd0: Log is configured to *not* fsync() on all Append() calls
I20260924 16:15:40.429216 13340 tablet_bootstrap.cc:492] T ce3789dc5b2e4b85aa79863bad2ebf34 P 12899fec5f5c4194af251b18585e0469: Bootstrap starting.
I20260924 16:15:40.429479 13338 tablet_bootstrap.cc:492] T ce3789dc5b2e4b85aa79863bad2ebf34 P f991653d74f04af180776405e08e2bd0: No bootstrap required, opened a new log
I20260924 16:15:40.429930 13338 ts_tablet_manager.cc:1397] T ce3789dc5b2e4b85aa79863bad2ebf34 P f991653d74f04af180776405e08e2bd0: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260924 16:15:40.431877 13342 tablet_bootstrap.cc:492] T ce3789dc5b2e4b85aa79863bad2ebf34 P 493291c4622e44f19e482afeb9191500: Bootstrap starting.
I20260924 16:15:40.432858 13340 tablet_bootstrap.cc:654] T ce3789dc5b2e4b85aa79863bad2ebf34 P 12899fec5f5c4194af251b18585e0469: Neither blocks nor log segments found. Creating new log.
I20260924 16:15:40.434270 13340 log.cc:824] T ce3789dc5b2e4b85aa79863bad2ebf34 P 12899fec5f5c4194af251b18585e0469: Log is configured to *not* fsync() on all Append() calls
I20260924 16:15:40.434595 13052 heartbeater.cc:503] Master 127.29.234.62:35061 was elected leader, sending a full tablet report...
I20260924 16:15:40.437510 13342 tablet_bootstrap.cc:654] T ce3789dc5b2e4b85aa79863bad2ebf34 P 493291c4622e44f19e482afeb9191500: Neither blocks nor log segments found. Creating new log.
I20260924 16:15:40.438879 13342 log.cc:824] T ce3789dc5b2e4b85aa79863bad2ebf34 P 493291c4622e44f19e482afeb9191500: Log is configured to *not* fsync() on all Append() calls
I20260924 16:15:40.440618 13338 raft_consensus.cc:348] T ce3789dc5b2e4b85aa79863bad2ebf34 P f991653d74f04af180776405e08e2bd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f991653d74f04af180776405e08e2bd0" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 42789 } } peers { permanent_uuid: "12899fec5f5c4194af251b18585e0469" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 38119 } } peers { permanent_uuid: "493291c4622e44f19e482afeb9191500" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41577 } }
I20260924 16:15:40.441071 13338 raft_consensus.cc:374] T ce3789dc5b2e4b85aa79863bad2ebf34 P f991653d74f04af180776405e08e2bd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:15:40.441238 13338 raft_consensus.cc:729] T ce3789dc5b2e4b85aa79863bad2ebf34 P f991653d74f04af180776405e08e2bd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f991653d74f04af180776405e08e2bd0, State: Initialized, Role: FOLLOWER
I20260924 16:15:40.441921 13338 consensus_queue.cc:260] T ce3789dc5b2e4b85aa79863bad2ebf34 P f991653d74f04af180776405e08e2bd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f991653d74f04af180776405e08e2bd0" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 42789 } } peers { permanent_uuid: "12899fec5f5c4194af251b18585e0469" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 38119 } } peers { permanent_uuid: "493291c4622e44f19e482afeb9191500" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41577 } }
I20260924 16:15:40.442332 13340 tablet_bootstrap.cc:492] T ce3789dc5b2e4b85aa79863bad2ebf34 P 12899fec5f5c4194af251b18585e0469: No bootstrap required, opened a new log
I20260924 16:15:40.442386 13342 tablet_bootstrap.cc:492] T ce3789dc5b2e4b85aa79863bad2ebf34 P 493291c4622e44f19e482afeb9191500: No bootstrap required, opened a new log
I20260924 16:15:40.442646 13340 ts_tablet_manager.cc:1397] T ce3789dc5b2e4b85aa79863bad2ebf34 P 12899fec5f5c4194af251b18585e0469: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.007s
I20260924 16:15:40.442776 13342 ts_tablet_manager.cc:1397] T ce3789dc5b2e4b85aa79863bad2ebf34 P 493291c4622e44f19e482afeb9191500: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20260924 16:15:40.443856 13338 ts_tablet_manager.cc:1428] T ce3789dc5b2e4b85aa79863bad2ebf34 P f991653d74f04af180776405e08e2bd0: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260924 16:15:40.450454 13342 raft_consensus.cc:348] T ce3789dc5b2e4b85aa79863bad2ebf34 P 493291c4622e44f19e482afeb9191500 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f991653d74f04af180776405e08e2bd0" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 42789 } } peers { permanent_uuid: "12899fec5f5c4194af251b18585e0469" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 38119 } } peers { permanent_uuid: "493291c4622e44f19e482afeb9191500" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41577 } }
I20260924 16:15:40.450826 13342 raft_consensus.cc:374] T ce3789dc5b2e4b85aa79863bad2ebf34 P 493291c4622e44f19e482afeb9191500 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:15:40.450930 13342 raft_consensus.cc:729] T ce3789dc5b2e4b85aa79863bad2ebf34 P 493291c4622e44f19e482afeb9191500 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 493291c4622e44f19e482afeb9191500, State: Initialized, Role: FOLLOWER
I20260924 16:15:40.451454 13342 consensus_queue.cc:260] T ce3789dc5b2e4b85aa79863bad2ebf34 P 493291c4622e44f19e482afeb9191500 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f991653d74f04af180776405e08e2bd0" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 42789 } } peers { permanent_uuid: "12899fec5f5c4194af251b18585e0469" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 38119 } } peers { permanent_uuid: "493291c4622e44f19e482afeb9191500" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41577 } }
I20260924 16:15:40.451687 13340 raft_consensus.cc:348] T ce3789dc5b2e4b85aa79863bad2ebf34 P 12899fec5f5c4194af251b18585e0469 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f991653d74f04af180776405e08e2bd0" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 42789 } } peers { permanent_uuid: "12899fec5f5c4194af251b18585e0469" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 38119 } } peers { permanent_uuid: "493291c4622e44f19e482afeb9191500" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41577 } }
I20260924 16:15:40.452150 13340 raft_consensus.cc:374] T ce3789dc5b2e4b85aa79863bad2ebf34 P 12899fec5f5c4194af251b18585e0469 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:15:40.452250 13340 raft_consensus.cc:729] T ce3789dc5b2e4b85aa79863bad2ebf34 P 12899fec5f5c4194af251b18585e0469 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12899fec5f5c4194af251b18585e0469, State: Initialized, Role: FOLLOWER
I20260924 16:15:40.452502 13187 heartbeater.cc:503] Master 127.29.234.62:35061 was elected leader, sending a full tablet report...
I20260924 16:15:40.452831 13340 consensus_queue.cc:260] T ce3789dc5b2e4b85aa79863bad2ebf34 P 12899fec5f5c4194af251b18585e0469 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f991653d74f04af180776405e08e2bd0" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 42789 } } peers { permanent_uuid: "12899fec5f5c4194af251b18585e0469" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 38119 } } peers { permanent_uuid: "493291c4622e44f19e482afeb9191500" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41577 } }
I20260924 16:15:40.453235 13342 ts_tablet_manager.cc:1428] T ce3789dc5b2e4b85aa79863bad2ebf34 P 493291c4622e44f19e482afeb9191500: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260924 16:15:40.453946 13322 heartbeater.cc:503] Master 127.29.234.62:35061 was elected leader, sending a full tablet report...
I20260924 16:15:40.454638 13340 ts_tablet_manager.cc:1428] T ce3789dc5b2e4b85aa79863bad2ebf34 P 12899fec5f5c4194af251b18585e0469: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260924 16:15:40.481441 13347 raft_consensus.cc:482] T ce3789dc5b2e4b85aa79863bad2ebf34 P 12899fec5f5c4194af251b18585e0469 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:15:40.481729 13347 raft_consensus.cc:504] T ce3789dc5b2e4b85aa79863bad2ebf34 P 12899fec5f5c4194af251b18585e0469 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f991653d74f04af180776405e08e2bd0" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 42789 } } peers { permanent_uuid: "12899fec5f5c4194af251b18585e0469" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 38119 } } peers { permanent_uuid: "493291c4622e44f19e482afeb9191500" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41577 } }
I20260924 16:15:40.483598 13347 leader_election.cc:288] T ce3789dc5b2e4b85aa79863bad2ebf34 P 12899fec5f5c4194af251b18585e0469 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f991653d74f04af180776405e08e2bd0 (127.29.234.1:42789), 493291c4622e44f19e482afeb9191500 (127.29.234.2:41577)
I20260924 16:15:40.489218 13324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:40.483977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.2:41577 (local address 127.29.234.3:58013)
0924 16:15:40.484169 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:40.484193 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:40.484309 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:40.484826 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:40.484831 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:40.484859 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:15:40.485036 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:40.485044 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:40.486102 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:40.486121 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:40.487395 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:40.487406 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:40.488479 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:40.488484 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:40.488663 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:40.488686 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 16:15:40.488791 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 16:15:40.488919 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260924 16:15:40.489207 13337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:40.484352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.3:58013 (local address 127.29.234.2:41577)
0924 16:15:40.484542 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:40.484546 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:40.484560 (+ 14us) server_negotiation.cc:407] Connection header received
0924 16:15:40.484617 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:40.484621 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:40.484689 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:40.484794 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:15:40.485177 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:40.485973 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:40.487556 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:40.488322 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:40.488478 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:40.488550 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:40.488822 (+ 272us) server_negotiation.cc:299] Negotiation successful
0924 16:15:40.488948 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 16:15:40.489986 13333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:40.484089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.3:40197 (local address 127.29.234.1:42789)
0924 16:15:40.484307 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:40.484315 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:40.484938 (+ 623us) server_negotiation.cc:407] Connection header received
0924 16:15:40.485138 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:40.485142 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:40.485205 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:40.485300 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:15:40.485819 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:40.486580 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:40.488235 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:40.488992 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:40.489165 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:40.489246 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:40.489636 (+ 390us) server_negotiation.cc:299] Negotiation successful
0924 16:15:40.489781 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 16:15:40.490074 13348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:40.483977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:42789 (local address 127.29.234.3:40197)
0924 16:15:40.484830 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:40.484843 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:40.485001 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:40.485314 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:40.485318 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:40.485341 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:15:40.485666 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:40.485673 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:40.486733 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:40.486744 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:40.488074 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:40.488085 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:40.489195 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:40.489202 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:40.489427 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:40.489458 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 16:15:40.489575 (+ 117us) client_negotiation.cc:240] Negotiation successful
0924 16:15:40.489688 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":724,"mutex_wait_us":79,"thread_start_us":447,"threads_started":1}
I20260924 16:15:40.490378 13139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3789dc5b2e4b85aa79863bad2ebf34" candidate_uuid: "12899fec5f5c4194af251b18585e0469" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "493291c4622e44f19e482afeb9191500" is_pre_election: true
I20260924 16:15:40.490916 13139 raft_consensus.cc:2454] T ce3789dc5b2e4b85aa79863bad2ebf34 P 493291c4622e44f19e482afeb9191500 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12899fec5f5c4194af251b18585e0469 in term 0.
I20260924 16:15:40.491201 13006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3789dc5b2e4b85aa79863bad2ebf34" candidate_uuid: "12899fec5f5c4194af251b18585e0469" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f991653d74f04af180776405e08e2bd0" is_pre_election: true
I20260924 16:15:40.491644 13207 leader_election.cc:302] T ce3789dc5b2e4b85aa79863bad2ebf34 P 12899fec5f5c4194af251b18585e0469 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12899fec5f5c4194af251b18585e0469, 493291c4622e44f19e482afeb9191500; no voters:
I20260924 16:15:40.491726 13006 raft_consensus.cc:2454] T ce3789dc5b2e4b85aa79863bad2ebf34 P f991653d74f04af180776405e08e2bd0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12899fec5f5c4194af251b18585e0469 in term 0.
I20260924 16:15:40.492067 13347 raft_consensus.cc:2781] T ce3789dc5b2e4b85aa79863bad2ebf34 P 12899fec5f5c4194af251b18585e0469 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:15:40.492236 13347 raft_consensus.cc:482] T ce3789dc5b2e4b85aa79863bad2ebf34 P 12899fec5f5c4194af251b18585e0469 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:15:40.492377 13347 raft_consensus.cc:3037] T ce3789dc5b2e4b85aa79863bad2ebf34 P 12899fec5f5c4194af251b18585e0469 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:15:40.494649 13347 raft_consensus.cc:504] T ce3789dc5b2e4b85aa79863bad2ebf34 P 12899fec5f5c4194af251b18585e0469 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f991653d74f04af180776405e08e2bd0" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 42789 } } peers { permanent_uuid: "12899fec5f5c4194af251b18585e0469" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 38119 } } peers { permanent_uuid: "493291c4622e44f19e482afeb9191500" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41577 } }
I20260924 16:15:40.495230 13347 leader_election.cc:288] T ce3789dc5b2e4b85aa79863bad2ebf34 P 12899fec5f5c4194af251b18585e0469 [CANDIDATE]: Term 1 election: Requested vote from peers f991653d74f04af180776405e08e2bd0 (127.29.234.1:42789), 493291c4622e44f19e482afeb9191500 (127.29.234.2:41577)
I20260924 16:15:40.495664 13006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3789dc5b2e4b85aa79863bad2ebf34" candidate_uuid: "12899fec5f5c4194af251b18585e0469" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f991653d74f04af180776405e08e2bd0"
I20260924 16:15:40.495747 13139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3789dc5b2e4b85aa79863bad2ebf34" candidate_uuid: "12899fec5f5c4194af251b18585e0469" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "493291c4622e44f19e482afeb9191500"
I20260924 16:15:40.495875 13006 raft_consensus.cc:3037] T ce3789dc5b2e4b85aa79863bad2ebf34 P f991653d74f04af180776405e08e2bd0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:15:40.495921 13139 raft_consensus.cc:3037] T ce3789dc5b2e4b85aa79863bad2ebf34 P 493291c4622e44f19e482afeb9191500 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:15:40.499112 13139 raft_consensus.cc:2454] T ce3789dc5b2e4b85aa79863bad2ebf34 P 493291c4622e44f19e482afeb9191500 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12899fec5f5c4194af251b18585e0469 in term 1.
I20260924 16:15:40.499112 13006 raft_consensus.cc:2454] T ce3789dc5b2e4b85aa79863bad2ebf34 P f991653d74f04af180776405e08e2bd0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12899fec5f5c4194af251b18585e0469 in term 1.
I20260924 16:15:40.499558 13206 leader_election.cc:302] T ce3789dc5b2e4b85aa79863bad2ebf34 P 12899fec5f5c4194af251b18585e0469 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12899fec5f5c4194af251b18585e0469, f991653d74f04af180776405e08e2bd0; no voters:
I20260924 16:15:40.499881 13347 raft_consensus.cc:2781] T ce3789dc5b2e4b85aa79863bad2ebf34 P 12899fec5f5c4194af251b18585e0469 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:15:40.500366 13347 raft_consensus.cc:686] T ce3789dc5b2e4b85aa79863bad2ebf34 P 12899fec5f5c4194af251b18585e0469 [term 1 LEADER]: Becoming Leader. State: Replica: 12899fec5f5c4194af251b18585e0469, State: Running, Role: LEADER
I20260924 16:15:40.500903 13347 consensus_queue.cc:237] T ce3789dc5b2e4b85aa79863bad2ebf34 P 12899fec5f5c4194af251b18585e0469 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f991653d74f04af180776405e08e2bd0" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 42789 } } peers { permanent_uuid: "12899fec5f5c4194af251b18585e0469" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 38119 } } peers { permanent_uuid: "493291c4622e44f19e482afeb9191500" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41577 } }
I20260924 16:15:40.505649 12863 catalog_manager.cc:5495] T ce3789dc5b2e4b85aa79863bad2ebf34 P 12899fec5f5c4194af251b18585e0469 reported cstate change: term changed from 0 to 1, leader changed from <none> to 12899fec5f5c4194af251b18585e0469 (127.29.234.3). New cstate: current_term: 1 leader_uuid: "12899fec5f5c4194af251b18585e0469" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f991653d74f04af180776405e08e2bd0" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 42789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12899fec5f5c4194af251b18585e0469" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 38119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "493291c4622e44f19e482afeb9191500" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41577 } health_report { overall_health: UNKNOWN } } }
I20260924 16:15:40.550403 13335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:40.545201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51582 (local address 127.29.234.3:38119)
0924 16:15:40.545449 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:40.545453 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:40.545472 (+ 19us) server_negotiation.cc:407] Connection header received
0924 16:15:40.545615 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:40.545629 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:40.545707 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:40.545833 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:15:40.546171 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:40.546866 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:40.548254 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:40.548630 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:40.548720 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:40.549265 (+ 545us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:15:40.549813 (+ 548us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:15:40.549933 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:40.550039 (+ 106us) server_negotiation.cc:299] Negotiation successful
0924 16:15:40.550101 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 16:15:40.558720 13139 raft_consensus.cc:1261] T ce3789dc5b2e4b85aa79863bad2ebf34 P 493291c4622e44f19e482afeb9191500 [term 1 FOLLOWER]: Refusing update from remote peer 12899fec5f5c4194af251b18585e0469: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:15:40.558741 13006 raft_consensus.cc:1261] T ce3789dc5b2e4b85aa79863bad2ebf34 P f991653d74f04af180776405e08e2bd0 [term 1 FOLLOWER]: Refusing update from remote peer 12899fec5f5c4194af251b18585e0469: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:15:40.559739 13349 consensus_queue.cc:1035] T ce3789dc5b2e4b85aa79863bad2ebf34 P 12899fec5f5c4194af251b18585e0469 [LEADER]: Connected to new peer: Peer: permanent_uuid: "493291c4622e44f19e482afeb9191500" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:15:40.560086 13347 consensus_queue.cc:1035] T ce3789dc5b2e4b85aa79863bad2ebf34 P 12899fec5f5c4194af251b18585e0469 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f991653d74f04af180776405e08e2bd0" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 42789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:15:40.584700 12862 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:36746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:15:40.584949 12862 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:36746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:15:40.588639 12862 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c32f836d405347ed9b2b10575a31a604: Sending DeleteTablet for 3 replicas of tablet ce3789dc5b2e4b85aa79863bad2ebf34
I20260924 16:15:40.589907 12986 tablet_service.cc:1514] Processing DeleteTablet for tablet ce3789dc5b2e4b85aa79863bad2ebf34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:15:40 UTC) from {username='kudu', principal='kudu/127.29.234.62@KRBTEST.COM'} at 127.0.0.1:52978
I20260924 16:15:40.589937 13256 tablet_service.cc:1514] Processing DeleteTablet for tablet ce3789dc5b2e4b85aa79863bad2ebf34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:15:40 UTC) from {username='kudu', principal='kudu/127.29.234.62@KRBTEST.COM'} at 127.0.0.1:51572
I20260924 16:15:40.590101 13119 tablet_service.cc:1514] Processing DeleteTablet for tablet ce3789dc5b2e4b85aa79863bad2ebf34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:15:40 UTC) from {username='kudu', principal='kudu/127.29.234.62@KRBTEST.COM'} at 127.0.0.1:46876
I20260924 16:15:40.591548 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 12901
I20260924 16:15:40.601763 13364 tablet_replica.cc:330] T ce3789dc5b2e4b85aa79863bad2ebf34 P 493291c4622e44f19e482afeb9191500: stopping tablet replica
I20260924 16:15:40.602468 13364 raft_consensus.cc:2229] T ce3789dc5b2e4b85aa79863bad2ebf34 P 493291c4622e44f19e482afeb9191500 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:15:40.603176 13364 raft_consensus.cc:2258] T ce3789dc5b2e4b85aa79863bad2ebf34 P 493291c4622e44f19e482afeb9191500 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:15:40.605970 13364 ts_tablet_manager.cc:1905] T ce3789dc5b2e4b85aa79863bad2ebf34 P 493291c4622e44f19e482afeb9191500: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:15:40.613878 13364 ts_tablet_manager.cc:1918] T ce3789dc5b2e4b85aa79863bad2ebf34 P 493291c4622e44f19e482afeb9191500: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:15:40.614600 13363 tablet_replica.cc:330] T ce3789dc5b2e4b85aa79863bad2ebf34 P f991653d74f04af180776405e08e2bd0: stopping tablet replica
I20260924 16:15:40.615233 13363 raft_consensus.cc:2229] T ce3789dc5b2e4b85aa79863bad2ebf34 P f991653d74f04af180776405e08e2bd0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:15:40.615859 13363 raft_consensus.cc:2258] T ce3789dc5b2e4b85aa79863bad2ebf34 P f991653d74f04af180776405e08e2bd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:15:40.616595 13364 log.cc:1196] T ce3789dc5b2e4b85aa79863bad2ebf34 P 493291c4622e44f19e482afeb9191500: Deleting WAL directory at /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-1/wal/wals/ce3789dc5b2e4b85aa79863bad2ebf34
I20260924 16:15:40.617595 13333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:40.592485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52992 (local address 127.29.234.1:42789)
0924 16:15:40.593217 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:40.593227 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:40.593250 (+ 23us) server_negotiation.cc:407] Connection header received
0924 16:15:40.593320 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:40.593329 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:40.593487 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:40.593603 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:40.596164 (+ 2561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:40.597683 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:40.600243 (+ 2560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:40.601164 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:40.601299 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:40.608204 (+ 6905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:40.608225 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:40.608244 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:15:40.608291 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:40.612345 (+ 4054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:40.613365 (+ 1020us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:40.613375 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:40.613378 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:40.613553 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:40.614791 (+ 1238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:40.614796 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:40.614799 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:40.615344 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:40.615644 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:40.616948 (+ 1304us) server_negotiation.cc:299] Negotiation successful
0924 16:15:40.617197 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563}
I20260924 16:15:40.618400 13047 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:15:40.618682 13362 tablet_replica.cc:330] T ce3789dc5b2e4b85aa79863bad2ebf34 P 12899fec5f5c4194af251b18585e0469: stopping tablet replica
I20260924 16:15:40.619088 13364 ts_tablet_manager.cc:1939] T ce3789dc5b2e4b85aa79863bad2ebf34 P 493291c4622e44f19e482afeb9191500: Deleting consensus metadata
I20260924 16:15:40.619151 13362 raft_consensus.cc:2229] T ce3789dc5b2e4b85aa79863bad2ebf34 P 12899fec5f5c4194af251b18585e0469 [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:15:40.619946 13362 raft_consensus.cc:2258] T ce3789dc5b2e4b85aa79863bad2ebf34 P 12899fec5f5c4194af251b18585e0469 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:15:40.621263 12848 catalog_manager.cc:4841] TS 493291c4622e44f19e482afeb9191500 (127.29.234.2:41577): tablet ce3789dc5b2e4b85aa79863bad2ebf34 (table test-table [id=01941355f0d8474fbe1b57e6318322fa]) successfully deleted
I20260924 16:15:40.623152 13362 ts_tablet_manager.cc:1905] T ce3789dc5b2e4b85aa79863bad2ebf34 P 12899fec5f5c4194af251b18585e0469: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:15:40.631071 13362 ts_tablet_manager.cc:1918] T ce3789dc5b2e4b85aa79863bad2ebf34 P 12899fec5f5c4194af251b18585e0469: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:15:40.631244 13362 log.cc:1196] T ce3789dc5b2e4b85aa79863bad2ebf34 P 12899fec5f5c4194af251b18585e0469: Deleting WAL directory at /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-2/wal/wals/ce3789dc5b2e4b85aa79863bad2ebf34
I20260924 16:15:40.631873 13362 ts_tablet_manager.cc:1939] T ce3789dc5b2e4b85aa79863bad2ebf34 P 12899fec5f5c4194af251b18585e0469: Deleting consensus metadata
I20260924 16:15:40.633671 12848 catalog_manager.cc:4841] TS 12899fec5f5c4194af251b18585e0469 (127.29.234.3:38119): tablet ce3789dc5b2e4b85aa79863bad2ebf34 (table test-table [id=01941355f0d8474fbe1b57e6318322fa]) successfully deleted
I20260924 16:15:40.618818 13363 ts_tablet_manager.cc:1905] T ce3789dc5b2e4b85aa79863bad2ebf34 P f991653d74f04af180776405e08e2bd0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:15:40.713970 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 12901
W20260924 16:15:40.732918 12847 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.234.1:42789 (error 108)
I20260924 16:15:40.733103 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 13056
I20260924 16:15:40.735263 13336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:40.734778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:42789 (local address 127.0.0.1:52994)
0924 16:15:40.735037 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:40.735128 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.1:42789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149}
W20260924 16:15:40.735525 12847 catalog_manager.cc:4584] TS f991653d74f04af180776405e08e2bd0 (127.29.234.1:42789): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ce3789dc5b2e4b85aa79863bad2ebf34: Network error: Client connection negotiation failed: client connection to 127.29.234.1:42789: connect: Connection refused (error 111)
I20260924 16:15:40.749899 13337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:40.734068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46886 (local address 127.29.234.2:41577)
0924 16:15:40.734262 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:40.734269 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:40.734380 (+ 111us) server_negotiation.cc:407] Connection header received
0924 16:15:40.734585 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:40.734593 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:40.734689 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:40.734807 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:40.736606 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:40.737581 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:40.739533 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:40.739919 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:40.740012 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:40.743495 (+ 3483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:40.743510 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:40.743528 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:15:40.743563 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:40.746709 (+ 3146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:40.747608 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:40.747617 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:40.747621 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:40.747752 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:40.748169 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:40.748173 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:40.748175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:40.748564 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:40.748792 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:40.749247 (+ 455us) server_negotiation.cc:299] Negotiation successful
0924 16:15:40.749509 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 16:15:40.750566 13182 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:15:40.789520 13336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:40.788876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:42789 (local address 127.0.0.1:52998)
0924 16:15:40.789137 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:40.789273 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.1:42789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260924 16:15:40.790762 13336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:40.790311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:42789 (local address 127.0.0.1:53004)
0924 16:15:40.790478 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:40.790607 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.1:42789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260924 16:15:40.842784 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 13056
I20260924 16:15:40.852237 13336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:40.851572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:42789 (local address 127.0.0.1:53020)
0924 16:15:40.851869 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:40.851989 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.1:42789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260924 16:15:40.853609 13336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:40.853080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:42789 (local address 127.0.0.1:53036)
0924 16:15:40.853266 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:40.853460 (+ 194us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.1:42789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260924 16:15:40.862632 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 13190
I20260924 16:15:40.878984 13335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:40.863418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51588 (local address 127.29.234.3:38119)
0924 16:15:40.863651 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:40.863657 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:40.863681 (+ 24us) server_negotiation.cc:407] Connection header received
0924 16:15:40.863794 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:40.863800 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:40.863888 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:40.863994 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:40.865786 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:40.867043 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:40.868396 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:40.869091 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:40.869244 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:40.871969 (+ 2725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:40.871983 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:40.872005 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:15:40.872054 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:40.876033 (+ 3979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:40.876768 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:40.876777 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:40.876780 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:40.876970 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:40.877326 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:40.877332 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:40.877335 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:40.877839 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:40.878156 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:40.878352 (+ 196us) server_negotiation.cc:299] Negotiation successful
0924 16:15:40.878563 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 16:15:40.879796 13317 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:15:40.934315 13336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:40.933804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:42789 (local address 127.0.0.1:53052)
0924 16:15:40.934030 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:40.934119 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.1:42789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260924 16:15:40.935491 13336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:40.935080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:42789 (local address 127.0.0.1:53054)
0924 16:15:40.935310 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:40.935380 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.1:42789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
I20260924 16:15:41.011593 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 13190
I20260924 16:15:41.031814 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 12830
I20260924 16:15:41.033094 12893 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:15:41.149677 13336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:41.148914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:42789 (local address 127.0.0.1:53062)
0924 16:15:41.149270 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:41.149437 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.1:42789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155}
I20260924 16:15:41.149749 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 12830
I20260924 16:15:41.172068 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:35061
--webserver_interface=127.29.234.62
--webserver_port=46067
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab
--principal=kudu/127.29.234.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:45085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.234.62:35061
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:15:41.514027 13370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:15:41.514321 13370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:15:41.514510 13370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:15:41.521845 13370 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:15:41.521955 13370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:15:41.522044 13370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:15:41.522152 13370 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:15:41.522236 13370 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:15:41.532212 13370 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:45085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.234.62:35061
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab
--principal=kudu/127.29.234.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:35061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.234.62
--webserver_port=46067
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:15:41.533717 13370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:15:41.535121 13370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:15:41.548058 13376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:41.548132 13375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:41.548734 13378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:15:41.548883 13370 server_base.cc:1034] running on GCE node
I20260924 16:15:41.549873 13370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:15:41.551431 13370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:15:41.552729 13370 hybrid_clock.cc:648] HybridClock initialized: now 1790266541552661 us; error 56 us; skew 500 ppm
I20260924 16:15:41.553138 13370 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:15:41 dist-test-slave-v5j9 krb5kdc[12269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266541, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:15:41.556617 13370 init.cc:377] Logged in from keytab as kudu/127.29.234.62@KRBTEST.COM (short username kudu)
I20260924 16:15:41.558156 13370 webserver.cc:460] Webserver started at http://127.29.234.62:46067/ using document root <none> and password file <none>
I20260924 16:15:41.558847 13370 fs_manager.cc:359] Metadata directory not provided
I20260924 16:15:41.558951 13370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:15:41.564582 13370 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260924 16:15:41.567471 13385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:41.568856 13370 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 16:15:41.568994 13370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/master-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/master-0/wal
uuid: "c32f836d405347ed9b2b10575a31a604"
format_stamp: "Formatted at 2026-09-24 16:15:35 on dist-test-slave-v5j9"
server_key: "534e502a9197cb96561efb4001391942"
server_key_iv: "3fe3b87a490785895b4bbba7ead71661"
server_key_version: "encryptionkey@0"
I20260924 16:15:41.570050 13370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:15:41.594916 13370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:15:41.599073 13370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:15:41.599364 13370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:15:41.618036 13370 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.62:35061
I20260924 16:15:41.618048 13437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.62:35061 every 8 connection(s)
I20260924 16:15:41.619601 13370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
I20260924 16:15:41.626626 13438 sys_catalog.cc:263] Verifying existing consensus state
I20260924 16:15:41.627328 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 13370
I20260924 16:15:41.628152 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.1:42789
--local_ip_for_outbound_sockets=127.29.234.1
--tserver_master_addrs=127.29.234.62:35061
--webserver_port=40655
--webserver_interface=127.29.234.1
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab
--principal=kudu/127.29.234.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:45085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:15:41.629985 13438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c32f836d405347ed9b2b10575a31a604: Bootstrap starting.
I20260924 16:15:41.650269 13438 log.cc:824] T 00000000000000000000000000000000 P c32f836d405347ed9b2b10575a31a604: Log is configured to *not* fsync() on all Append() calls
I20260924 16:15:41.666692 13438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c32f836d405347ed9b2b10575a31a604: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 16:15:41.667220 13438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c32f836d405347ed9b2b10575a31a604: Bootstrap complete.
I20260924 16:15:41.676692 13438 raft_consensus.cc:348] T 00000000000000000000000000000000 P c32f836d405347ed9b2b10575a31a604 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c32f836d405347ed9b2b10575a31a604" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 35061 } }
I20260924 16:15:41.677699 13438 raft_consensus.cc:729] T 00000000000000000000000000000000 P c32f836d405347ed9b2b10575a31a604 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: c32f836d405347ed9b2b10575a31a604, State: Initialized, Role: FOLLOWER
I20260924 16:15:41.678313 13438 consensus_queue.cc:260] T 00000000000000000000000000000000 P c32f836d405347ed9b2b10575a31a604 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c32f836d405347ed9b2b10575a31a604" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 35061 } }
I20260924 16:15:41.678562 13438 raft_consensus.cc:388] T 00000000000000000000000000000000 P c32f836d405347ed9b2b10575a31a604 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:15:41.678691 13438 raft_consensus.cc:482] T 00000000000000000000000000000000 P c32f836d405347ed9b2b10575a31a604 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:15:41.678874 13438 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c32f836d405347ed9b2b10575a31a604 [term 2 FOLLOWER]: Advancing to term 3
I20260924 16:15:41.682909 13438 raft_consensus.cc:504] T 00000000000000000000000000000000 P c32f836d405347ed9b2b10575a31a604 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c32f836d405347ed9b2b10575a31a604" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 35061 } }
I20260924 16:15:41.683418 13438 leader_election.cc:302] T 00000000000000000000000000000000 P c32f836d405347ed9b2b10575a31a604 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c32f836d405347ed9b2b10575a31a604; no voters:
I20260924 16:15:41.684029 13438 leader_election.cc:288] T 00000000000000000000000000000000 P c32f836d405347ed9b2b10575a31a604 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 16:15:41.684223 13442 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c32f836d405347ed9b2b10575a31a604 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 16:15:41.685434 13442 raft_consensus.cc:686] T 00000000000000000000000000000000 P c32f836d405347ed9b2b10575a31a604 [term 3 LEADER]: Becoming Leader. State: Replica: c32f836d405347ed9b2b10575a31a604, State: Running, Role: LEADER
I20260924 16:15:41.686244 13438 sys_catalog.cc:564] T 00000000000000000000000000000000 P c32f836d405347ed9b2b10575a31a604 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:15:41.686086 13442 consensus_queue.cc:237] T 00000000000000000000000000000000 P c32f836d405347ed9b2b10575a31a604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c32f836d405347ed9b2b10575a31a604" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 35061 } }
I20260924 16:15:41.688822 13444 sys_catalog.cc:455] T 00000000000000000000000000000000 P c32f836d405347ed9b2b10575a31a604 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c32f836d405347ed9b2b10575a31a604. Latest consensus state: current_term: 3 leader_uuid: "c32f836d405347ed9b2b10575a31a604" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c32f836d405347ed9b2b10575a31a604" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 35061 } } }
I20260924 16:15:41.689159 13444 sys_catalog.cc:458] T 00000000000000000000000000000000 P c32f836d405347ed9b2b10575a31a604 [sys.catalog]: This master's current role is: LEADER
I20260924 16:15:41.689713 13443 sys_catalog.cc:455] T 00000000000000000000000000000000 P c32f836d405347ed9b2b10575a31a604 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "c32f836d405347ed9b2b10575a31a604" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c32f836d405347ed9b2b10575a31a604" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 35061 } } }
I20260924 16:15:41.689978 13443 sys_catalog.cc:458] T 00000000000000000000000000000000 P c32f836d405347ed9b2b10575a31a604 [sys.catalog]: This master's current role is: LEADER
I20260924 16:15:41.702484 13449 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:15:41.709772 13449 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:15:41.710896 13449 catalog_manager.cc:1259] Loaded cluster ID: 5f4cda6912f7450594aa256ddbe1ceb6
I20260924 16:15:41.711014 13449 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260924 16:15:41.712497 13449 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f6004a879ad google::LogMessage::Fail() at ??:0
@ 0x7f6004a8a88a google::LogMessage::SendToLog() at ??:0
@ 0x7f6004a874de google::LogMessage::Flush() at ??:0
@ 0x7f6004a88a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f601a4ba512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f601a4b7976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f601a53f610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f601a53f439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f60076d3345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f60076d691c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f60076d6799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f60076b1eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f600a98b6db start_thread at ??:0
@ 0x7f600392161f clone at ??:0
W20260924 16:15:41.999863 13440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:15:42.000231 13440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:15:42.000408 13440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:15:42.008329 13440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:15:42.008507 13440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:15:42.008692 13440 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.1
I20260924 16:15:42.019003 13440 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:45085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab
--principal=kudu/127.29.234.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.1:42789
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.234.1
--webserver_port=40655
--tserver_master_addrs=127.29.234.62:35061
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.1
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:15:42.020653 13440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:15:42.022087 13440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:15:42.035701 13480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:42.035758 13483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:42.035784 13481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:15:42.036691 13440 server_base.cc:1034] running on GCE node
I20260924 16:15:42.037262 13440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:15:42.038322 13440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:15:42.039502 13440 hybrid_clock.cc:648] HybridClock initialized: now 1790266542039470 us; error 43 us; skew 500 ppm
I20260924 16:15:42.039865 13440 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:15:42 dist-test-slave-v5j9 krb5kdc[12269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266542, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:15:42.043036 13440 init.cc:377] Logged in from keytab as kudu/127.29.234.1@KRBTEST.COM (short username kudu)
I20260924 16:15:42.044526 13440 webserver.cc:460] Webserver started at http://127.29.234.1:40655/ using document root <none> and password file <none>
I20260924 16:15:42.045241 13440 fs_manager.cc:359] Metadata directory not provided
I20260924 16:15:42.045429 13440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:15:42.051172 13440 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 16:15:42.054281 13490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:42.055716 13440 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 16:15:42.055918 13440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-0/wal
uuid: "f991653d74f04af180776405e08e2bd0"
format_stamp: "Formatted at 2026-09-24 16:15:36 on dist-test-slave-v5j9"
server_key: "c8b280f8598c43a400bc4fdeb8e2648f"
server_key_iv: "0d2a977e2b1d6e86b92f43b018962069"
server_key_version: "encryptionkey@0"
I20260924 16:15:42.056934 13440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:15:42.076915 13440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:15:42.081575 13440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:15:42.081868 13440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:15:42.082890 13440 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:15:42.084741 13498 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260924 16:15:42.091755 13440 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 16:15:42.091915 13440 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.008s user 0.000s sys 0.000s
I20260924 16:15:42.092074 13440 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 16:15:42.095407 13440 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 16:15:42.095517 13440 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.002s sys 0.000s
I20260924 16:15:42.096040 13498 tablet_bootstrap.cc:492] T ce3789dc5b2e4b85aa79863bad2ebf34 P f991653d74f04af180776405e08e2bd0: Bootstrap starting.
I20260924 16:15:42.105453 13498 log.cc:824] T ce3789dc5b2e4b85aa79863bad2ebf34 P f991653d74f04af180776405e08e2bd0: Log is configured to *not* fsync() on all Append() calls
I20260924 16:15:42.124922 13498 tablet_bootstrap.cc:492] T ce3789dc5b2e4b85aa79863bad2ebf34 P f991653d74f04af180776405e08e2bd0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 16:15:42.125821 13498 tablet_bootstrap.cc:492] T ce3789dc5b2e4b85aa79863bad2ebf34 P f991653d74f04af180776405e08e2bd0: Bootstrap complete.
I20260924 16:15:42.126562 13498 ts_tablet_manager.cc:1397] T ce3789dc5b2e4b85aa79863bad2ebf34 P f991653d74f04af180776405e08e2bd0: Time spent bootstrapping tablet: real 0.031s user 0.029s sys 0.000s
I20260924 16:15:42.134608 13440 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.1:42789
I20260924 16:15:42.134692 13606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.1:42789 every 8 connection(s)
I20260924 16:15:42.136317 13440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
I20260924 16:15:42.139046 13498 raft_consensus.cc:348] T ce3789dc5b2e4b85aa79863bad2ebf34 P f991653d74f04af180776405e08e2bd0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f991653d74f04af180776405e08e2bd0" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 42789 } } peers { permanent_uuid: "12899fec5f5c4194af251b18585e0469" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 38119 } } peers { permanent_uuid: "493291c4622e44f19e482afeb9191500" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41577 } }
I20260924 16:15:42.140540 13498 raft_consensus.cc:729] T ce3789dc5b2e4b85aa79863bad2ebf34 P f991653d74f04af180776405e08e2bd0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f991653d74f04af180776405e08e2bd0, State: Initialized, Role: FOLLOWER
I20260924 16:15:42.141332 13498 consensus_queue.cc:260] T ce3789dc5b2e4b85aa79863bad2ebf34 P f991653d74f04af180776405e08e2bd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f991653d74f04af180776405e08e2bd0" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 42789 } } peers { permanent_uuid: "12899fec5f5c4194af251b18585e0469" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 38119 } } peers { permanent_uuid: "493291c4622e44f19e482afeb9191500" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41577 } }
I20260924 16:15:42.141810 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 13440
I20260924 16:15:42.142477 13609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:42.139707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:35061 (local address 127.29.234.1:34903)
0924 16:15:42.140814 (+ 1107us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:42.141064 (+ 250us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:35061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":706,"thread_start_us":396,"threads_started":1}
I20260924 16:15:42.142582 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.2:41577
--local_ip_for_outbound_sockets=127.29.234.2
--tserver_master_addrs=127.29.234.62:35061
--webserver_port=44075
--webserver_interface=127.29.234.2
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab
--principal=kudu/127.29.234.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:45085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:15:42.143054 13498 ts_tablet_manager.cc:1428] T ce3789dc5b2e4b85aa79863bad2ebf34 P f991653d74f04af180776405e08e2bd0: Time spent starting tablet: real 0.016s user 0.004s sys 0.012s
W20260924 16:15:42.143237 13607 heartbeater.cc:644] Failed to heartbeat to 127.29.234.62:35061 (0 consecutive failures): Network error: Failed to ping master at 127.29.234.62:35061: Client connection negotiation failed: client connection to 127.29.234.62:35061: connect: Connection refused (error 111)
I20260924 16:15:42.144181 13609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:42.143742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:35061 (local address 127.29.234.1:46965)
0924 16:15:42.143929 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:42.144024 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:35061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260924 16:15:42.145067 13609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:42.144648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:35061 (local address 127.29.234.1:38475)
0924 16:15:42.144817 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:42.144911 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:35061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260924 16:15:42.145342 13607 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 16:15:42.476475 13611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:15:42.476754 13611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:15:42.476922 13611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:15:42.484206 13611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:15:42.484318 13611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:15:42.484515 13611 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.2
I20260924 16:15:42.494484 13611 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:45085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab
--principal=kudu/127.29.234.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.2:41577
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.234.2
--webserver_port=44075
--tserver_master_addrs=127.29.234.62:35061
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.2
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:15:42.495893 13611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:15:42.497288 13611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:15:42.510182 13616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:42.510687 13617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:42.510768 13619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:15:42.511372 13611 server_base.cc:1034] running on GCE node
I20260924 16:15:42.511943 13611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:15:42.512826 13611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:15:42.514072 13611 hybrid_clock.cc:648] HybridClock initialized: now 1790266542514012 us; error 62 us; skew 500 ppm
I20260924 16:15:42.514421 13611 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:15:42 dist-test-slave-v5j9 krb5kdc[12269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266542, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:15:42.517777 13611 init.cc:377] Logged in from keytab as kudu/127.29.234.2@KRBTEST.COM (short username kudu)
I20260924 16:15:42.519251 13611 webserver.cc:460] Webserver started at http://127.29.234.2:44075/ using document root <none> and password file <none>
I20260924 16:15:42.519977 13611 fs_manager.cc:359] Metadata directory not provided
I20260924 16:15:42.520128 13611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:15:42.525691 13611 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260924 16:15:42.528297 13626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:42.529667 13611 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:15:42.529857 13611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-1/wal
uuid: "493291c4622e44f19e482afeb9191500"
format_stamp: "Formatted at 2026-09-24 16:15:36 on dist-test-slave-v5j9"
server_key: "277fe4ba9eb1034086a741e02efada08"
server_key_iv: "a93b1f0c7143d992a7e5f8fb6e984152"
server_key_version: "encryptionkey@0"
I20260924 16:15:42.530825 13611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:15:42.545181 13611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:15:42.549222 13611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:15:42.549527 13611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:15:42.550482 13611 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:15:42.551812 13611 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:15:42.551941 13611 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:42.552070 13611 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:15:42.552160 13611 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:42.586642 13611 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.2:41577
I20260924 16:15:42.586701 13740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.2:41577 every 8 connection(s)
I20260924 16:15:42.588131 13611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
I20260924 16:15:42.593070 13743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:42.591244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:35061 (local address 127.29.234.2:43757)
0924 16:15:42.592001 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:42.592226 (+ 225us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:35061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":306,"threads_started":1}
W20260924 16:15:42.593811 13741 heartbeater.cc:644] Failed to heartbeat to 127.29.234.62:35061 (0 consecutive failures): Network error: Failed to ping master at 127.29.234.62:35061: Client connection negotiation failed: client connection to 127.29.234.62:35061: connect: Connection refused (error 111)
I20260924 16:15:42.594599 13743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:42.594300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:35061 (local address 127.29.234.2:44647)
0924 16:15:42.594434 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:42.594499 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:35061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260924 16:15:42.595382 13743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:42.595076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:35061 (local address 127.29.234.2:42487)
0924 16:15:42.595204 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:42.595276 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:35061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260924 16:15:42.595643 13741 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 16:15:42.596894 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 13611
I20260924 16:15:42.597599 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.3:38119
--local_ip_for_outbound_sockets=127.29.234.3
--tserver_master_addrs=127.29.234.62:35061
--webserver_port=44797
--webserver_interface=127.29.234.3
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab
--principal=kudu/127.29.234.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:45085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:15:42.925257 13744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:15:42.925654 13744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:15:42.925761 13744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:15:42.933239 13744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:15:42.933380 13744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:15:42.933552 13744 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.3
I20260924 16:15:42.943662 13744 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:45085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab
--principal=kudu/127.29.234.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.3:38119
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.234.3
--webserver_port=44797
--tserver_master_addrs=127.29.234.62:35061
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.3
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:15:42.945039 13744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:15:42.946625 13744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:15:42.960304 13749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:42.960685 13752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:15:42.960757 13750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:15:42.962404 13744 server_base.cc:1034] running on GCE node
I20260924 16:15:42.963164 13744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:15:42.964267 13744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:15:42.965458 13744 hybrid_clock.cc:648] HybridClock initialized: now 1790266542965407 us; error 36 us; skew 500 ppm
I20260924 16:15:42.965817 13744 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:15:42 dist-test-slave-v5j9 krb5kdc[12269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266542, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:15:42.969118 13744 init.cc:377] Logged in from keytab as kudu/127.29.234.3@KRBTEST.COM (short username kudu)
I20260924 16:15:42.970825 13744 webserver.cc:460] Webserver started at http://127.29.234.3:44797/ using document root <none> and password file <none>
I20260924 16:15:42.971598 13744 fs_manager.cc:359] Metadata directory not provided
I20260924 16:15:42.971758 13744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:15:42.977564 13744 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260924 16:15:42.980650 13759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:42.982263 13744 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:15:42.982448 13744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-2/wal
uuid: "12899fec5f5c4194af251b18585e0469"
format_stamp: "Formatted at 2026-09-24 16:15:37 on dist-test-slave-v5j9"
server_key: "6754491885e551fa0c9efa254bbfbcd9"
server_key_iv: "d977910b03c2242e1b2b3f4495f600b1"
server_key_version: "encryptionkey@0"
I20260924 16:15:42.983541 13744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:15:42.995229 13744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:15:42.999500 13744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:15:42.999788 13744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:15:43.000816 13744 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:15:43.002195 13744 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:15:43.002314 13744 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:43.002447 13744 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:15:43.002537 13744 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:15:43.036247 13744 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.3:38119
I20260924 16:15:43.036340 13873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.3:38119 every 8 connection(s)
I20260924 16:15:43.037886 13744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
I20260924 16:15:43.040489 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 13744
I20260924 16:15:43.043902 13877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:43.041568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:35061 (local address 127.29.234.3:55157)
0924 16:15:43.042761 (+ 1193us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:43.042985 (+ 224us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:35061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":771,"thread_start_us":543,"threads_started":1}
W20260924 16:15:43.044749 13874 heartbeater.cc:644] Failed to heartbeat to 127.29.234.62:35061 (0 consecutive failures): Network error: Failed to ping master at 127.29.234.62:35061: Client connection negotiation failed: client connection to 127.29.234.62:35061: connect: Connection refused (error 111)
I20260924 16:15:43.045753 13877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:43.045288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:35061 (local address 127.29.234.3:54125)
0924 16:15:43.045518 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:43.045592 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:35061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260924 16:15:43.046639 13877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:43.046295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:35061 (local address 127.29.234.3:59091)
0924 16:15:43.046431 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:43.046508 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:35061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260924 16:15:43.046918 13874 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 16:15:43.146612 13878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:43.145841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:35061 (local address 127.29.234.1:51883)
0924 16:15:43.146307 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:43.146420 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:35061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":218,"threads_started":1}
I20260924 16:15:43.324676 13879 raft_consensus.cc:482] T ce3789dc5b2e4b85aa79863bad2ebf34 P f991653d74f04af180776405e08e2bd0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:15:43.325101 13879 raft_consensus.cc:504] T ce3789dc5b2e4b85aa79863bad2ebf34 P f991653d74f04af180776405e08e2bd0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f991653d74f04af180776405e08e2bd0" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 42789 } } peers { permanent_uuid: "12899fec5f5c4194af251b18585e0469" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 38119 } } peers { permanent_uuid: "493291c4622e44f19e482afeb9191500" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41577 } }
I20260924 16:15:43.326809 13879 leader_election.cc:288] T ce3789dc5b2e4b85aa79863bad2ebf34 P f991653d74f04af180776405e08e2bd0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12899fec5f5c4194af251b18585e0469 (127.29.234.3:38119), 493291c4622e44f19e482afeb9191500 (127.29.234.2:41577)
Sep 24 16:15:43 dist-test-slave-v5j9 krb5kdc[12269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266542, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.1@KRBTEST.COM for kudu/127.29.234.3@KRBTEST.COM
Sep 24 16:15:43 dist-test-slave-v5j9 krb5kdc[12269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266542, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.1@KRBTEST.COM for kudu/127.29.234.2@KRBTEST.COM
I20260924 16:15:43.350915 13878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:43.327234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.3:38119 (local address 127.29.234.1:39569)
0924 16:15:43.327480 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:43.327522 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:43.328243 (+ 721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:43.332248 (+ 4005us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:43.332259 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:43.333102 (+ 843us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:15:43.334192 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:43.334215 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:43.335963 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:43.335966 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:43.337857 (+ 1891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:43.337868 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:43.338851 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:43.338858 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:43.339114 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:43.340003 (+ 889us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:15:43.340029 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:15:43.342970 (+ 2941us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:15:43.347476 (+ 4506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:15:43.347483 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:15:43.347497 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:15:43.347937 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:15:43.348433 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:15:43.348436 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:15:43.348439 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:15:43.348596 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:15:43.349541 (+ 945us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:15:43.349552 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:15:43.349892 (+ 340us) client_negotiation.cc:769] Sending connection context
0924 16:15:43.350224 (+ 332us) client_negotiation.cc:240] Negotiation successful
0924 16:15:43.350577 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67,"mutex_wait_us":53}
I20260924 16:15:43.351342 13880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:43.327504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.1:39569 (local address 127.29.234.3:38119)
0924 16:15:43.329171 (+ 1667us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:43.329188 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:43.329225 (+ 37us) server_negotiation.cc:407] Connection header received
0924 16:15:43.331334 (+ 2109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:43.331344 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:43.331750 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:43.332100 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:43.334395 (+ 2295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:43.335765 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:43.337997 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:43.338646 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:43.338858 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:43.343285 (+ 4427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:43.343292 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:43.343307 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:15:43.343344 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:43.347333 (+ 3989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:43.348084 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:43.348094 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:43.348102 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:43.348317 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:43.348717 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:43.348722 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:43.348726 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:43.349385 (+ 659us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:43.349577 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:43.350500 (+ 923us) server_negotiation.cc:299] Negotiation successful
0924 16:15:43.350916 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1490,"thread_start_us":1319,"threads_started":1}
I20260924 16:15:43.351929 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:43.327658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.2:41577 (local address 127.29.234.1:42259)
0924 16:15:43.333603 (+ 5945us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:43.333622 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 16:15:43.333812 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:15:43.336412 (+ 2600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:15:43.336417 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:15:43.337081 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:15:43.337471 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:43.337484 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:43.339112 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:43.339118 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:43.340410 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:15:43.340423 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:15:43.341503 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:15:43.341509 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:15:43.341621 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:43.342216 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:15:43.342238 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:15:43.344969 (+ 2731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:15:43.349069 (+ 4100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:15:43.349072 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:15:43.349074 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:15:43.349494 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:15:43.349911 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:15:43.349919 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:15:43.349925 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:15:43.350106 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:15:43.351177 (+ 1071us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:15:43.351181 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:15:43.351333 (+ 152us) client_negotiation.cc:769] Sending connection context
0924 16:15:43.351472 (+ 139us) client_negotiation.cc:240] Negotiation successful
0924 16:15:43.351694 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5809,"mutex_wait_us":1,"thread_start_us":5623,"threads_started":1}
I20260924 16:15:43.353143 13882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:43.328047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.1:42259 (local address 127.29.234.2:41577)
0924 16:15:43.333088 (+ 5041us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:43.333102 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:43.333747 (+ 645us) server_negotiation.cc:407] Connection header received
0924 16:15:43.335718 (+ 1971us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:43.335726 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:43.336080 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:43.336408 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:43.337645 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:43.338886 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:43.340578 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:43.341314 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:43.341506 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:43.345283 (+ 3777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:43.345298 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:43.345316 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:15:43.345385 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:43.348936 (+ 3551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:43.349631 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:43.349642 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:43.349648 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:43.349798 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:43.350262 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:43.350270 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:43.350276 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:43.350923 (+ 647us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:43.351094 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:43.352339 (+ 1245us) server_negotiation.cc:299] Negotiation successful
0924 16:15:43.352742 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4849,"thread_start_us":4721,"threads_started":1}
W20260924 16:15:43.354743 13493 leader_election.cc:341] T ce3789dc5b2e4b85aa79863bad2ebf34 P f991653d74f04af180776405e08e2bd0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 12899fec5f5c4194af251b18585e0469 (127.29.234.3:38119): Not found: Tablet not found: ce3789dc5b2e4b85aa79863bad2ebf34
W20260924 16:15:43.355666 13493 leader_election.cc:341] T ce3789dc5b2e4b85aa79863bad2ebf34 P f991653d74f04af180776405e08e2bd0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 493291c4622e44f19e482afeb9191500 (127.29.234.2:41577): Not found: Tablet not found: ce3789dc5b2e4b85aa79863bad2ebf34
I20260924 16:15:43.355821 13493 leader_election.cc:302] T ce3789dc5b2e4b85aa79863bad2ebf34 P f991653d74f04af180776405e08e2bd0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f991653d74f04af180776405e08e2bd0; no voters: 12899fec5f5c4194af251b18585e0469, 493291c4622e44f19e482afeb9191500
I20260924 16:15:43.356253 13879 raft_consensus.cc:2726] T ce3789dc5b2e4b85aa79863bad2ebf34 P f991653d74f04af180776405e08e2bd0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 16:15:43.597018 13883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:43.596181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:35061 (local address 127.29.234.2:39537)
0924 16:15:43.596645 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:43.596791 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:35061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":214,"threads_started":1}
I20260924 16:15:44.048219 13884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:44.047441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:35061 (local address 127.29.234.3:39501)
0924 16:15:44.047884 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:44.047984 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:35061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":213,"threads_started":1}
I20260924 16:15:44.148311 13885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:44.147531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:35061 (local address 127.29.234.1:56999)
0924 16:15:44.148050 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:44.148144 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:35061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":308,"threads_started":1}
I20260924 16:15:44.598634 13886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:44.597970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:35061 (local address 127.29.234.2:53785)
0924 16:15:44.598387 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:44.598469 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:35061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":224,"threads_started":1}
I20260924 16:15:45.049952 13887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:45.049123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:35061 (local address 127.29.234.3:43135)
0924 16:15:45.049667 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:45.049751 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:35061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":321,"threads_started":1}
I20260924 16:15:45.150388 13888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:45.149478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:35061 (local address 127.29.234.1:45813)
0924 16:15:45.150118 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:45.150208 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:35061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":317,"threads_started":1}
I20260924 16:15:45.249620 13889 raft_consensus.cc:482] T ce3789dc5b2e4b85aa79863bad2ebf34 P f991653d74f04af180776405e08e2bd0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:15:45.249846 13889 raft_consensus.cc:504] T ce3789dc5b2e4b85aa79863bad2ebf34 P f991653d74f04af180776405e08e2bd0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f991653d74f04af180776405e08e2bd0" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 42789 } } peers { permanent_uuid: "12899fec5f5c4194af251b18585e0469" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 38119 } } peers { permanent_uuid: "493291c4622e44f19e482afeb9191500" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41577 } }
I20260924 16:15:45.250576 13889 leader_election.cc:288] T ce3789dc5b2e4b85aa79863bad2ebf34 P f991653d74f04af180776405e08e2bd0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12899fec5f5c4194af251b18585e0469 (127.29.234.3:38119), 493291c4622e44f19e482afeb9191500 (127.29.234.2:41577)
W20260924 16:15:45.251652 13493 leader_election.cc:341] T ce3789dc5b2e4b85aa79863bad2ebf34 P f991653d74f04af180776405e08e2bd0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 12899fec5f5c4194af251b18585e0469 (127.29.234.3:38119): Not found: Tablet not found: ce3789dc5b2e4b85aa79863bad2ebf34
W20260924 16:15:45.251920 13493 leader_election.cc:341] T ce3789dc5b2e4b85aa79863bad2ebf34 P f991653d74f04af180776405e08e2bd0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 493291c4622e44f19e482afeb9191500 (127.29.234.2:41577): Not found: Tablet not found: ce3789dc5b2e4b85aa79863bad2ebf34
I20260924 16:15:45.252041 13493 leader_election.cc:302] T ce3789dc5b2e4b85aa79863bad2ebf34 P f991653d74f04af180776405e08e2bd0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f991653d74f04af180776405e08e2bd0; no voters: 12899fec5f5c4194af251b18585e0469, 493291c4622e44f19e482afeb9191500
I20260924 16:15:45.252374 13889 raft_consensus.cc:2726] T ce3789dc5b2e4b85aa79863bad2ebf34 P f991653d74f04af180776405e08e2bd0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 16:15:45.600365 13890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:45.599587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:35061 (local address 127.29.234.2:46553)
0924 16:15:45.600089 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:45.600182 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:35061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":255,"threads_started":1}
I20260924 16:15:46.051461 13891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:46.050792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:35061 (local address 127.29.234.3:48609)
0924 16:15:46.051234 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:46.051317 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:35061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":220,"threads_started":1}
I20260924 16:15:46.151993 13892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:46.151282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:35061 (local address 127.29.234.1:43867)
0924 16:15:46.151730 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:46.151825 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:35061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":236,"threads_started":1}
I20260924 16:15:46.602253 13893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:46.601495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:35061 (local address 127.29.234.2:37605)
0924 16:15:46.601939 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:46.602026 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:35061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":213,"threads_started":1}
I20260924 16:15:47.053390 13894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:47.052456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:35061 (local address 127.29.234.3:60797)
0924 16:15:47.053034 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:47.053125 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:35061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":262,"threads_started":1}
I20260924 16:15:47.154096 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:47.153084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:35061 (local address 127.29.234.1:46477)
0924 16:15:47.153803 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:47.153894 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:35061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":310,"threads_started":1}
I20260924 16:15:47.603914 13896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:47.603189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:35061 (local address 127.29.234.2:37983)
0924 16:15:47.603652 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:47.603737 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:35061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":235,"threads_started":1}
I20260924 16:15:47.668813 13897 raft_consensus.cc:482] T ce3789dc5b2e4b85aa79863bad2ebf34 P f991653d74f04af180776405e08e2bd0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:15:47.669035 13897 raft_consensus.cc:504] T ce3789dc5b2e4b85aa79863bad2ebf34 P f991653d74f04af180776405e08e2bd0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f991653d74f04af180776405e08e2bd0" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 42789 } } peers { permanent_uuid: "12899fec5f5c4194af251b18585e0469" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 38119 } } peers { permanent_uuid: "493291c4622e44f19e482afeb9191500" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41577 } }
I20260924 16:15:47.669816 13897 leader_election.cc:288] T ce3789dc5b2e4b85aa79863bad2ebf34 P f991653d74f04af180776405e08e2bd0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12899fec5f5c4194af251b18585e0469 (127.29.234.3:38119), 493291c4622e44f19e482afeb9191500 (127.29.234.2:41577)
W20260924 16:15:47.670737 13493 leader_election.cc:341] T ce3789dc5b2e4b85aa79863bad2ebf34 P f991653d74f04af180776405e08e2bd0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 12899fec5f5c4194af251b18585e0469 (127.29.234.3:38119): Not found: Tablet not found: ce3789dc5b2e4b85aa79863bad2ebf34
W20260924 16:15:47.670950 13493 leader_election.cc:341] T ce3789dc5b2e4b85aa79863bad2ebf34 P f991653d74f04af180776405e08e2bd0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 493291c4622e44f19e482afeb9191500 (127.29.234.2:41577): Not found: Tablet not found: ce3789dc5b2e4b85aa79863bad2ebf34
I20260924 16:15:47.671077 13493 leader_election.cc:302] T ce3789dc5b2e4b85aa79863bad2ebf34 P f991653d74f04af180776405e08e2bd0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f991653d74f04af180776405e08e2bd0; no voters: 12899fec5f5c4194af251b18585e0469, 493291c4622e44f19e482afeb9191500
I20260924 16:15:47.671366 13897 raft_consensus.cc:2726] T ce3789dc5b2e4b85aa79863bad2ebf34 P f991653d74f04af180776405e08e2bd0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 16:15:48.055119 13898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:48.054427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:35061 (local address 127.29.234.3:47093)
0924 16:15:48.054875 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:48.054961 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:35061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":238,"threads_started":1}
I20260924 16:15:48.155810 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:48.155043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:35061 (local address 127.29.234.1:39821)
0924 16:15:48.155513 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:48.155612 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:35061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":253,"threads_started":1}
I20260924 16:15:48.605602 13900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:48.604747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:35061 (local address 127.29.234.2:54239)
0924 16:15:48.605260 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:48.605401 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:35061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":249,"threads_started":1}
I20260924 16:15:49.056859 13901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:49.056093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:35061 (local address 127.29.234.3:48479)
0924 16:15:49.056572 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:49.056658 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:35061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":256,"threads_started":1}
I20260924 16:15:49.157521 13902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:49.156736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:35061 (local address 127.29.234.1:34109)
0924 16:15:49.157180 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:49.157265 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:35061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":238,"threads_started":1}
I20260924 16:15:49.607335 13903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:49.606548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:35061 (local address 127.29.234.2:59971)
0924 16:15:49.607067 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:49.607152 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:35061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":241,"threads_started":1}
I20260924 16:15:50.058781 13904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:50.057897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:35061 (local address 127.29.234.3:50347)
0924 16:15:50.058440 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:50.058532 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:35061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":252,"threads_started":1}
I20260924 16:15:50.159134 13905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:50.158425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:35061 (local address 127.29.234.1:51745)
0924 16:15:50.158863 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:50.158958 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:35061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":246,"threads_started":1}
I20260924 16:15:50.609067 13906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:50.608349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:35061 (local address 127.29.234.2:49985)
0924 16:15:50.608784 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:50.608867 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:35061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":225,"threads_started":1}
I20260924 16:15:50.877976 13907 raft_consensus.cc:482] T ce3789dc5b2e4b85aa79863bad2ebf34 P f991653d74f04af180776405e08e2bd0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:15:50.878225 13907 raft_consensus.cc:504] T ce3789dc5b2e4b85aa79863bad2ebf34 P f991653d74f04af180776405e08e2bd0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f991653d74f04af180776405e08e2bd0" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 42789 } } peers { permanent_uuid: "12899fec5f5c4194af251b18585e0469" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 38119 } } peers { permanent_uuid: "493291c4622e44f19e482afeb9191500" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41577 } }
I20260924 16:15:50.878873 13907 leader_election.cc:288] T ce3789dc5b2e4b85aa79863bad2ebf34 P f991653d74f04af180776405e08e2bd0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12899fec5f5c4194af251b18585e0469 (127.29.234.3:38119), 493291c4622e44f19e482afeb9191500 (127.29.234.2:41577)
W20260924 16:15:50.879892 13493 leader_election.cc:341] T ce3789dc5b2e4b85aa79863bad2ebf34 P f991653d74f04af180776405e08e2bd0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 12899fec5f5c4194af251b18585e0469 (127.29.234.3:38119): Not found: Tablet not found: ce3789dc5b2e4b85aa79863bad2ebf34
W20260924 16:15:50.880182 13493 leader_election.cc:341] T ce3789dc5b2e4b85aa79863bad2ebf34 P f991653d74f04af180776405e08e2bd0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 493291c4622e44f19e482afeb9191500 (127.29.234.2:41577): Not found: Tablet not found: ce3789dc5b2e4b85aa79863bad2ebf34
I20260924 16:15:50.880294 13493 leader_election.cc:302] T ce3789dc5b2e4b85aa79863bad2ebf34 P f991653d74f04af180776405e08e2bd0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f991653d74f04af180776405e08e2bd0; no voters: 12899fec5f5c4194af251b18585e0469, 493291c4622e44f19e482afeb9191500
I20260924 16:15:50.880589 13907 raft_consensus.cc:2726] T ce3789dc5b2e4b85aa79863bad2ebf34 P f991653d74f04af180776405e08e2bd0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 16:15:51.060560 13908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:51.059756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:35061 (local address 127.29.234.3:35039)
0924 16:15:51.060216 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:51.060376 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:35061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":237,"threads_started":1}
I20260924 16:15:51.160957 13909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:51.160070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:35061 (local address 127.29.234.1:47507)
0924 16:15:51.160633 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:51.160737 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:35061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":246,"threads_started":1}
I20260924 16:15:51.611011 13910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:51.610254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:35061 (local address 127.29.234.2:35913)
0924 16:15:51.610764 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:51.610851 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:35061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":232,"threads_started":1}
I20260924 16:15:52.062767 13911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:52.061779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:35061 (local address 127.29.234.3:47801)
0924 16:15:52.062404 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:52.062520 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:35061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":298,"threads_started":1}
I20260924 16:15:52.162789 13912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:52.162022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:35061 (local address 127.29.234.1:35513)
0924 16:15:52.162541 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:52.162624 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:35061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":308,"threads_started":1}
I20260924 16:15:52.612754 13913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:52.611979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:35061 (local address 127.29.234.2:50301)
0924 16:15:52.612471 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:52.612557 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:35061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":273,"threads_started":1}
I20260924 16:15:53.064476 13914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:53.063792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:35061 (local address 127.29.234.3:36325)
0924 16:15:53.064216 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:53.064296 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:35061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":223,"threads_started":1}
I20260924 16:15:53.164399 13915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:53.163675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:35061 (local address 127.29.234.1:46729)
0924 16:15:53.164111 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:53.164219 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:35061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":233,"threads_started":1}
I20260924 16:15:53.614496 13916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:53.613769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:35061 (local address 127.29.234.2:35025)
0924 16:15:53.614248 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:53.614334 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:35061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":233,"threads_started":1}
I20260924 16:15:54.066228 13917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:54.065471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:35061 (local address 127.29.234.3:57143)
0924 16:15:54.065954 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:54.066038 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:35061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":232,"threads_started":1}
I20260924 16:15:54.166214 13918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:54.165459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:35061 (local address 127.29.234.1:60119)
0924 16:15:54.165926 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:54.166014 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:35061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":255,"threads_started":1}
I20260924 16:15:54.616320 13919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:54.615519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:35061 (local address 127.29.234.2:44733)
0924 16:15:54.616036 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:54.616129 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:35061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":259,"threads_started":1}
I20260924 16:15:55.068099 13920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:55.067268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:35061 (local address 127.29.234.3:37183)
0924 16:15:55.067823 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:55.067924 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:35061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":294,"threads_started":1}
I20260924 16:15:55.167992 13921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:55.167088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:35061 (local address 127.29.234.1:48333)
0924 16:15:55.167644 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:55.167770 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:35061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":266,"threads_started":1}
I20260924 16:15:55.351888 13922 raft_consensus.cc:482] T ce3789dc5b2e4b85aa79863bad2ebf34 P f991653d74f04af180776405e08e2bd0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:15:55.352164 13922 raft_consensus.cc:504] T ce3789dc5b2e4b85aa79863bad2ebf34 P f991653d74f04af180776405e08e2bd0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f991653d74f04af180776405e08e2bd0" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 42789 } } peers { permanent_uuid: "12899fec5f5c4194af251b18585e0469" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 38119 } } peers { permanent_uuid: "493291c4622e44f19e482afeb9191500" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 41577 } }
I20260924 16:15:55.352869 13922 leader_election.cc:288] T ce3789dc5b2e4b85aa79863bad2ebf34 P f991653d74f04af180776405e08e2bd0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12899fec5f5c4194af251b18585e0469 (127.29.234.3:38119), 493291c4622e44f19e482afeb9191500 (127.29.234.2:41577)
W20260924 16:15:55.353971 13493 leader_election.cc:341] T ce3789dc5b2e4b85aa79863bad2ebf34 P f991653d74f04af180776405e08e2bd0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 12899fec5f5c4194af251b18585e0469 (127.29.234.3:38119): Not found: Tablet not found: ce3789dc5b2e4b85aa79863bad2ebf34
W20260924 16:15:55.354203 13493 leader_election.cc:341] T ce3789dc5b2e4b85aa79863bad2ebf34 P f991653d74f04af180776405e08e2bd0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 493291c4622e44f19e482afeb9191500 (127.29.234.2:41577): Not found: Tablet not found: ce3789dc5b2e4b85aa79863bad2ebf34
I20260924 16:15:55.354328 13493 leader_election.cc:302] T ce3789dc5b2e4b85aa79863bad2ebf34 P f991653d74f04af180776405e08e2bd0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f991653d74f04af180776405e08e2bd0; no voters: 12899fec5f5c4194af251b18585e0469, 493291c4622e44f19e482afeb9191500
I20260924 16:15:55.354624 13922 raft_consensus.cc:2726] T ce3789dc5b2e4b85aa79863bad2ebf34 P f991653d74f04af180776405e08e2bd0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 16:15:55.618157 13923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:55.617339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:35061 (local address 127.29.234.2:56323)
0924 16:15:55.617867 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:55.617969 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:35061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":225,"threads_started":1}
I20260924 16:15:56.070024 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:56.069161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:35061 (local address 127.29.234.3:51715)
0924 16:15:56.069757 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:56.069854 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:35061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":347,"threads_started":1}
I20260924 16:15:56.169852 13925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:56.168923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:35061 (local address 127.29.234.1:55043)
0924 16:15:56.169493 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:56.169651 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:35061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":301,"threads_started":1}
I20260924 16:15:56.619925 13926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:56.619154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:35061 (local address 127.29.234.2:56465)
0924 16:15:56.619622 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:56.619709 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:35061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":228,"threads_started":1}
I20260924 16:15:57.071846 13927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:57.071034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:35061 (local address 127.29.234.3:58479)
0924 16:15:57.071527 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:57.071641 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:35061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":269,"threads_started":1}
I20260924 16:15:57.171859 13928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:57.171049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:35061 (local address 127.29.234.1:40683)
0924 16:15:57.171562 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:57.171682 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:35061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":250,"threads_started":1}
I20260924 16:15:57.622290 13929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:57.621331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:35061 (local address 127.29.234.2:34873)
0924 16:15:57.621939 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:57.622107 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:35061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":317,"threads_started":1}
I20260924 16:15:58.049901 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 13440
I20260924 16:15:58.074093 13931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:58.073173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:35061 (local address 127.29.234.3:55193)
0924 16:15:58.073758 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0924 16:15:58.073880 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:35061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":299,"threads_started":1}
I20260924 16:15:58.074440 13930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:58.051142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38544 (local address 127.29.234.1:42789)
0924 16:15:58.051773 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:58.051780 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:58.051809 (+ 29us) server_negotiation.cc:407] Connection header received
0924 16:15:58.051872 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:58.051879 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:58.052109 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:58.052250 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:58.053759 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:58.054730 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:58.055942 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:58.056577 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:58.056674 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:58.067964 (+ 11290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:58.067979 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:58.068000 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:15:58.068047 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:58.070990 (+ 2943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:58.071852 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:58.071860 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:58.071863 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:58.071994 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:58.072408 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:58.072414 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:58.072417 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:58.072943 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:58.073178 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:58.073747 (+ 569us) server_negotiation.cc:299] Negotiation successful
0924 16:15:58.074069 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":286,"threads_started":1}
I20260924 16:15:58.075752 13602 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:15:58.191725 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 13440
I20260924 16:15:58.221037 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 13611
I20260924 16:15:58.244995 13934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:58.222263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55632 (local address 127.29.234.2:41577)
0924 16:15:58.223126 (+ 863us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:58.223139 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:58.223166 (+ 27us) server_negotiation.cc:407] Connection header received
0924 16:15:58.223262 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:58.223267 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:58.223377 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:58.223605 (+ 228us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:58.226534 (+ 2929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:58.227828 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:58.229911 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:58.230654 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:58.230817 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:58.235355 (+ 4538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:58.235361 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:58.235364 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:15:58.235420 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:58.240321 (+ 4901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:58.241583 (+ 1262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:58.241589 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:58.241592 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:58.241816 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:58.242389 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:58.242392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:58.242394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:58.242994 (+ 600us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:58.243455 (+ 461us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:58.244030 (+ 575us) server_negotiation.cc:299] Negotiation successful
0924 16:15:58.244340 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":716,"thread_start_us":477,"threads_started":1}
I20260924 16:15:58.246224 13736 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:15:58.368672 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 13611
I20260924 16:15:58.397506 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 13744
I20260924 16:15:58.421836 13936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:15:58.398923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34912 (local address 127.29.234.3:38119)
0924 16:15:58.399923 (+ 1000us) server_negotiation.cc:206] Beginning negotiation
0924 16:15:58.399938 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 16:15:58.399966 (+ 28us) server_negotiation.cc:407] Connection header received
0924 16:15:58.400058 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:15:58.400062 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:15:58.400177 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:15:58.400344 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:15:58.402771 (+ 2427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:58.404095 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:58.406854 (+ 2759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:15:58.407843 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:15:58.408007 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:15:58.413005 (+ 4998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:15:58.413012 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:15:58.413015 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:15:58.413076 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:15:58.417822 (+ 4746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:58.418934 (+ 1112us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:58.418939 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:58.418941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:58.419170 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:15:58.419791 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:15:58.419794 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:15:58.419796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:15:58.420263 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:15:58.420570 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0924 16:15:58.421039 (+ 469us) server_negotiation.cc:299] Negotiation successful
0924 16:15:58.421393 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":795,"thread_start_us":547,"threads_started":1}
I20260924 16:15:58.423092 13869 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:15:58.545495 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 13744
2026-09-24T16:15:58Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24654 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:15:58 dist-test-slave-v5j9 krb5kdc[13940](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:15:58 dist-test-slave-v5j9 krb5kdc[13940](info): set up 2 sockets
Sep 24 16:15:58 dist-test-slave-v5j9 krb5kdc[13940](info): commencing operation
krb5kdc: starting...
W20260924 16:15:59.789582 30632 mini_kdc.cc:121] Time spent starting KDC: real 1.184s user 0.000s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Sep 24 16:15:59 dist-test-slave-v5j9 krb5kdc[13940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:15:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:15:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.234.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab.
Entry for principal kudu/127.29.234.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab.
I20260924 16:16:00.041311 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:44387
--webserver_interface=127.29.234.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab
--principal=kudu/127.29.234.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:45871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.234.62:44387
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:16:00.379042 13953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:16:00.379400 13953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:16:00.379568 13953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:16:00.386868 13953 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 16:16:00.387008 13953 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 16:16:00.387095 13953 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 16:16:00.387198 13953 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:16:00.387301 13953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:16:00.387418 13953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:16:00.387502 13953 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:16:00.387583 13953 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:16:00.397486 13953 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:45871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.29.234.62:44387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab
--principal=kudu/127.29.234.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:44387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.234.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:16:00.399195 13953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:16:00.400624 13953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:16:00.413017 13959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:16:00.412981 13960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:16:00.413213 13962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:16:00.414026 13953 server_base.cc:1034] running on GCE node
I20260924 16:16:00.414927 13953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:16:00.416364 13953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:16:00.417593 13953 hybrid_clock.cc:648] HybridClock initialized: now 1790266560417528 us; error 28 us; skew 500 ppm
I20260924 16:16:00.417970 13953 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:16:00 dist-test-slave-v5j9 krb5kdc[13940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266560, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:16:00.421108 13953 init.cc:377] Logged in from keytab as kudu/127.29.234.62@KRBTEST.COM (short username kudu)
I20260924 16:16:00.422727 13953 webserver.cc:460] Webserver started at http://127.29.234.62:41217/ using document root <none> and password file <none>
I20260924 16:16:00.423492 13953 fs_manager.cc:359] Metadata directory not provided
I20260924 16:16:00.423606 13953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:16:00.423950 13953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:16:00.427134 13953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/master-0/data/instance:
uuid: "92a389a3e45944bf836899bbf5e85366"
format_stamp: "Formatted at 2026-09-24 16:16:00 on dist-test-slave-v5j9"
server_key: "cc5bc5cfa98f40f77611023dcbd525df"
server_key_iv: "e93ec137cc70329d4350bc2009d92071"
server_key_version: "encryptionkey@0"
I20260924 16:16:00.427984 13953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/master-0/wal/instance:
uuid: "92a389a3e45944bf836899bbf5e85366"
format_stamp: "Formatted at 2026-09-24 16:16:00 on dist-test-slave-v5j9"
server_key: "cc5bc5cfa98f40f77611023dcbd525df"
server_key_iv: "e93ec137cc70329d4350bc2009d92071"
server_key_version: "encryptionkey@0"
I20260924 16:16:00.433593 13953 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 16:16:00.436992 13969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:16:00.438361 13953 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:16:00.438550 13953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/master-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/master-0/wal
uuid: "92a389a3e45944bf836899bbf5e85366"
format_stamp: "Formatted at 2026-09-24 16:16:00 on dist-test-slave-v5j9"
server_key: "cc5bc5cfa98f40f77611023dcbd525df"
server_key_iv: "e93ec137cc70329d4350bc2009d92071"
server_key_version: "encryptionkey@0"
I20260924 16:16:00.438784 13953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:16:00.454277 13953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:16:00.458357 13953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:16:00.458637 13953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:16:00.477054 13953 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.62:44387
I20260924 16:16:00.477078 14023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.62:44387 every 8 connection(s)
I20260924 16:16:00.478507 13953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
I20260924 16:16:00.482344 14024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:16:00.484462 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 13953
I20260924 16:16:00.484680 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/master-0/wal/instance
I20260924 16:16:00.485532 30632 external_mini_cluster.cc:1428] Setting key e671efe583a56add5c3b2817e1ff0ff5
I20260924 16:16:00.495643 14024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92a389a3e45944bf836899bbf5e85366: Bootstrap starting.
I20260924 16:16:00.499847 14024 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92a389a3e45944bf836899bbf5e85366: Neither blocks nor log segments found. Creating new log.
I20260924 16:16:00.501149 14024 log.cc:824] T 00000000000000000000000000000000 P 92a389a3e45944bf836899bbf5e85366: Log is configured to *not* fsync() on all Append() calls
Sep 24 16:16:00 dist-test-slave-v5j9 krb5kdc[13940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:16:00.504091 14024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92a389a3e45944bf836899bbf5e85366: No bootstrap required, opened a new log
I20260924 16:16:00.509944 14027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:00.487685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54924 (local address 127.29.234.62:44387)
0924 16:16:00.488511 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:00.488523 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:00.488568 (+ 45us) server_negotiation.cc:407] Connection header received
0924 16:16:00.490334 (+ 1766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:00.490360 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:00.490764 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:00.491095 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:00.493151 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:00.494571 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:00.496463 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:00.497158 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:00.497314 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:00.502319 (+ 5005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:16:00.502328 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:16:00.502343 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:16:00.502377 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:16:00.505680 (+ 3303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:00.506435 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:00.506442 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:00.506449 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:00.506617 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:00.507013 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:00.507018 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:00.507022 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:00.507590 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:16:00.507902 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:00.508420 (+ 518us) server_negotiation.cc:299] Negotiation successful
0924 16:16:00.508833 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":402,"threads_started":1}
I20260924 16:16:00.512642 14024 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92a389a3e45944bf836899bbf5e85366 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92a389a3e45944bf836899bbf5e85366" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 44387 } }
I20260924 16:16:00.513137 14024 raft_consensus.cc:374] T 00000000000000000000000000000000 P 92a389a3e45944bf836899bbf5e85366 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:16:00.513247 14024 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92a389a3e45944bf836899bbf5e85366 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92a389a3e45944bf836899bbf5e85366, State: Initialized, Role: FOLLOWER
I20260924 16:16:00.513835 14024 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92a389a3e45944bf836899bbf5e85366 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92a389a3e45944bf836899bbf5e85366" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 44387 } }
I20260924 16:16:00.514036 14024 raft_consensus.cc:388] T 00000000000000000000000000000000 P 92a389a3e45944bf836899bbf5e85366 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:16:00.514132 14024 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92a389a3e45944bf836899bbf5e85366 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:16:00.514250 14024 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92a389a3e45944bf836899bbf5e85366 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:16:00.516394 14024 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92a389a3e45944bf836899bbf5e85366 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92a389a3e45944bf836899bbf5e85366" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 44387 } }
I20260924 16:16:00.517010 14024 leader_election.cc:302] T 00000000000000000000000000000000 P 92a389a3e45944bf836899bbf5e85366 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92a389a3e45944bf836899bbf5e85366; no voters:
I20260924 16:16:00.517675 14024 leader_election.cc:288] T 00000000000000000000000000000000 P 92a389a3e45944bf836899bbf5e85366 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:16:00.517791 14029 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92a389a3e45944bf836899bbf5e85366 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:16:00.518476 14029 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92a389a3e45944bf836899bbf5e85366 [term 1 LEADER]: Becoming Leader. State: Replica: 92a389a3e45944bf836899bbf5e85366, State: Running, Role: LEADER
I20260924 16:16:00.519053 14029 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92a389a3e45944bf836899bbf5e85366 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92a389a3e45944bf836899bbf5e85366" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 44387 } }
I20260924 16:16:00.519778 14024 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92a389a3e45944bf836899bbf5e85366 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:16:00.522200 14030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92a389a3e45944bf836899bbf5e85366 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "92a389a3e45944bf836899bbf5e85366" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92a389a3e45944bf836899bbf5e85366" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 44387 } } }
I20260924 16:16:00.522733 14030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92a389a3e45944bf836899bbf5e85366 [sys.catalog]: This master's current role is: LEADER
I20260924 16:16:00.522200 14031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92a389a3e45944bf836899bbf5e85366 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92a389a3e45944bf836899bbf5e85366. Latest consensus state: current_term: 1 leader_uuid: "92a389a3e45944bf836899bbf5e85366" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92a389a3e45944bf836899bbf5e85366" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 44387 } } }
I20260924 16:16:00.523639 14031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92a389a3e45944bf836899bbf5e85366 [sys.catalog]: This master's current role is: LEADER
I20260924 16:16:00.527797 14036 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:16:00.532930 14036 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:16:00.540810 14036 catalog_manager.cc:1347] Generated new cluster ID: fe4623367c664b69928c0d8cd63e4002
I20260924 16:16:00.540918 14036 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:16:00.555153 14036 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:16:00.556493 14036 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:16:00.569643 14036 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 92a389a3e45944bf836899bbf5e85366: Generated new TSK 0
I20260924 16:16:00.570593 14036 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.234.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab.
Entry for principal kudu/127.29.234.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab.
I20260924 16:16:00.640545 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.1:0
--local_ip_for_outbound_sockets=127.29.234.1
--webserver_interface=127.29.234.1
--webserver_port=0
--tserver_master_addrs=127.29.234.62:44387
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab
--principal=kudu/127.29.234.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:45871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:16:00.982734 14050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:16:00.983110 14050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:16:00.983301 14050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:16:00.990597 14050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:16:00.990715 14050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:16:00.990929 14050 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.1
I20260924 16:16:01.000857 14050 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:45871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab
--principal=kudu/127.29.234.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.234.1
--webserver_port=0
--tserver_master_addrs=127.29.234.62:44387
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.1
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:16:01.002239 14050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:16:01.003741 14050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:16:01.017773 14059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:16:01.017815 14057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:16:01.017845 14056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:16:01.018617 14050 server_base.cc:1034] running on GCE node
I20260924 16:16:01.019330 14050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:16:01.020514 14050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:16:01.021811 14050 hybrid_clock.cc:648] HybridClock initialized: now 1790266561021739 us; error 54 us; skew 500 ppm
I20260924 16:16:01.022176 14050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:16:01 dist-test-slave-v5j9 krb5kdc[13940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266561, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:16:01.025882 14050 init.cc:377] Logged in from keytab as kudu/127.29.234.1@KRBTEST.COM (short username kudu)
I20260924 16:16:01.027267 14050 webserver.cc:460] Webserver started at http://127.29.234.1:43803/ using document root <none> and password file <none>
I20260924 16:16:01.027967 14050 fs_manager.cc:359] Metadata directory not provided
I20260924 16:16:01.028069 14050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:16:01.028331 14050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:16:01.031917 14050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-0/data/instance:
uuid: "4287e1c923de4b69a9c02d00cf413ae7"
format_stamp: "Formatted at 2026-09-24 16:16:01 on dist-test-slave-v5j9"
server_key: "d473dca69abee598abdf3d378d4d362d"
server_key_iv: "9608e050948fd8d14b4b6cfda4253373"
server_key_version: "encryptionkey@0"
I20260924 16:16:01.033049 14050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-0/wal/instance:
uuid: "4287e1c923de4b69a9c02d00cf413ae7"
format_stamp: "Formatted at 2026-09-24 16:16:01 on dist-test-slave-v5j9"
server_key: "d473dca69abee598abdf3d378d4d362d"
server_key_iv: "9608e050948fd8d14b4b6cfda4253373"
server_key_version: "encryptionkey@0"
I20260924 16:16:01.040441 14050 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.004s
I20260924 16:16:01.044224 14066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:16:01.045634 14050 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:16:01.045809 14050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-0/wal
uuid: "4287e1c923de4b69a9c02d00cf413ae7"
format_stamp: "Formatted at 2026-09-24 16:16:01 on dist-test-slave-v5j9"
server_key: "d473dca69abee598abdf3d378d4d362d"
server_key_iv: "9608e050948fd8d14b4b6cfda4253373"
server_key_version: "encryptionkey@0"
I20260924 16:16:01.046010 14050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:16:01.059504 14050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:16:01.063849 14050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:16:01.064133 14050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:16:01.065102 14050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:16:01.066357 14050 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:16:01.066429 14050 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:16:01.066509 14050 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:16:01.066556 14050 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:16:01.099985 14050 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.1:34979
I20260924 16:16:01.100032 14180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.1:34979 every 8 connection(s)
I20260924 16:16:01.101411 14050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
I20260924 16:16:01.105401 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 14050
I20260924 16:16:01.105638 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-0/wal/instance
I20260924 16:16:01.106359 30632 external_mini_cluster.cc:1428] Setting key fe59f68cb094cfb281f5171da7671c07
Sep 24 16:16:01 dist-test-slave-v5j9 krb5kdc[13940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266561, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.1@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:16:01.123780 14183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:01.104153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.1:40447 (local address 127.29.234.62:44387)
0924 16:16:01.104600 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:01.104605 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:01.106098 (+ 1493us) server_negotiation.cc:407] Connection header received
0924 16:16:01.107909 (+ 1811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:01.107915 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:01.108009 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:01.108127 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:01.109809 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:01.110541 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:01.112112 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:01.112528 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:01.112662 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:01.116219 (+ 3557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:16:01.116227 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:16:01.116235 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:16:01.116288 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:16:01.119989 (+ 3701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:01.120843 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:01.120851 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:01.120857 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:01.121009 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:01.121536 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:01.121544 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:01.121550 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:01.121941 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:16:01.122102 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:01.122971 (+ 869us) server_negotiation.cc:299] Negotiation successful
0924 16:16:01.123241 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":232,"threads_started":1}
I20260924 16:16:01.124567 14184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:01.104763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:44387 (local address 127.29.234.1:40447)
0924 16:16:01.105614 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:01.105661 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 16:16:01.107638 (+ 1977us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:16:01.108398 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:16:01.108407 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:16:01.108940 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:16:01.109626 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:16:01.109646 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:16:01.110719 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:16:01.110724 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:16:01.111961 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:16:01.111973 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:16:01.112774 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:16:01.112782 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:16:01.113018 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:01.113769 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:16:01.113794 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:16:01.116010 (+ 2216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:16:01.120133 (+ 4123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:16:01.120140 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:16:01.120155 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:16:01.120709 (+ 554us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:16:01.121126 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:16:01.121130 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:16:01.121132 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:16:01.121408 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:16:01.122163 (+ 755us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:16:01.122170 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:16:01.122542 (+ 372us) client_negotiation.cc:769] Sending connection context
0924 16:16:01.122791 (+ 249us) client_negotiation.cc:240] Negotiation successful
0924 16:16:01.123260 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":339,"threads_started":1}
I20260924 16:16:01.126408 14181 heartbeater.cc:347] Connected to a master server at 127.29.234.62:44387
I20260924 16:16:01.126773 14181 heartbeater.cc:464] Registering TS with master...
I20260924 16:16:01.127712 14181 heartbeater.cc:511] Master 127.29.234.62:44387 requested a full tablet report, sending...
I20260924 16:16:01.130475 13987 ts_manager.cc:194] Registered new tserver with Master: 4287e1c923de4b69a9c02d00cf413ae7 (127.29.234.1:34979)
WARNING: no policy specified for kudu/127.29.234.2@KRBTEST.COM; defaulting to no policy
I20260924 16:16:01.132776 13987 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.234.1@KRBTEST.COM'} at 127.29.234.1:40447
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab.
Entry for principal kudu/127.29.234.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab.
I20260924 16:16:01.164840 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.2:0
--local_ip_for_outbound_sockets=127.29.234.2
--webserver_interface=127.29.234.2
--webserver_port=0
--tserver_master_addrs=127.29.234.62:44387
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab
--principal=kudu/127.29.234.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:45871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:16:01.494165 14187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:16:01.494452 14187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:16:01.494616 14187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:16:01.502480 14187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:16:01.502592 14187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:16:01.502799 14187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.2
I20260924 16:16:01.517571 14187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:45871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab
--principal=kudu/127.29.234.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.234.2
--webserver_port=0
--tserver_master_addrs=127.29.234.62:44387
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.2
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:16:01.519616 14187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:16:01.521739 14187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:16:01.539078 14194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:16:01.539902 14193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:16:01.540457 14196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:16:01.540858 14187 server_base.cc:1034] running on GCE node
I20260924 16:16:01.541607 14187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:16:01.542783 14187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:16:01.544034 14187 hybrid_clock.cc:648] HybridClock initialized: now 1790266561543949 us; error 78 us; skew 500 ppm
I20260924 16:16:01.544390 14187 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:16:01 dist-test-slave-v5j9 krb5kdc[13940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266561, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:16:01.548741 14187 init.cc:377] Logged in from keytab as kudu/127.29.234.2@KRBTEST.COM (short username kudu)
I20260924 16:16:01.550514 14187 webserver.cc:460] Webserver started at http://127.29.234.2:45995/ using document root <none> and password file <none>
I20260924 16:16:01.551230 14187 fs_manager.cc:359] Metadata directory not provided
I20260924 16:16:01.551368 14187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:16:01.551807 14187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:16:01.555341 14187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-1/data/instance:
uuid: "ad7a4fa907754ea2849edb3d6089728f"
format_stamp: "Formatted at 2026-09-24 16:16:01 on dist-test-slave-v5j9"
server_key: "a01b3e512cad3af12263c7e771a4c6c3"
server_key_iv: "19a2275bc8b410cee47e3aa9552f924b"
server_key_version: "encryptionkey@0"
I20260924 16:16:01.556346 14187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-1/wal/instance:
uuid: "ad7a4fa907754ea2849edb3d6089728f"
format_stamp: "Formatted at 2026-09-24 16:16:01 on dist-test-slave-v5j9"
server_key: "a01b3e512cad3af12263c7e771a4c6c3"
server_key_iv: "19a2275bc8b410cee47e3aa9552f924b"
server_key_version: "encryptionkey@0"
I20260924 16:16:01.561575 14187 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260924 16:16:01.565186 14203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:16:01.566617 14187 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:16:01.566799 14187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-1/wal
uuid: "ad7a4fa907754ea2849edb3d6089728f"
format_stamp: "Formatted at 2026-09-24 16:16:01 on dist-test-slave-v5j9"
server_key: "a01b3e512cad3af12263c7e771a4c6c3"
server_key_iv: "19a2275bc8b410cee47e3aa9552f924b"
server_key_version: "encryptionkey@0"
I20260924 16:16:01.567018 14187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:16:01.586125 14187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:16:01.590541 14187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:16:01.590844 14187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:16:01.591863 14187 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:16:01.593221 14187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:16:01.593322 14187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:16:01.593497 14187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:16:01.593570 14187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:16:01.627766 14187 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.2:37037
I20260924 16:16:01.627806 14316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.2:37037 every 8 connection(s)
I20260924 16:16:01.629128 14187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
I20260924 16:16:01.629495 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 14187
I20260924 16:16:01.629676 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-1/wal/instance
I20260924 16:16:01.630250 30632 external_mini_cluster.cc:1428] Setting key 8a31147b068710db0849edcd5b8eece9
Sep 24 16:16:01 dist-test-slave-v5j9 krb5kdc[13940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266561, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.2@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:16:01.651291 14319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:01.631905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.2:52301 (local address 127.29.234.62:44387)
0924 16:16:01.632386 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:01.632393 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:01.633562 (+ 1169us) server_negotiation.cc:407] Connection header received
0924 16:16:01.635410 (+ 1848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:01.635419 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:01.635513 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:01.635631 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:01.637157 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:01.638377 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:01.639952 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:01.640390 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:01.640526 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:01.645045 (+ 4519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:16:01.645050 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:16:01.645054 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:16:01.645088 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:16:01.647804 (+ 2716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:01.648735 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:01.648738 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:01.648741 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:01.648868 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:01.649434 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:01.649438 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:01.649441 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:01.649748 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:16:01.649873 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:01.650825 (+ 952us) server_negotiation.cc:299] Negotiation successful
0924 16:16:01.650998 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":262,"threads_started":1}
I20260924 16:16:01.653055 14320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:01.632566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:44387 (local address 127.29.234.2:52301)
0924 16:16:01.633403 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:01.633447 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 16:16:01.635128 (+ 1681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:16:01.635841 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:16:01.635849 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:16:01.636402 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:16:01.636978 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:16:01.636996 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:16:01.638516 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:16:01.638520 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:16:01.639787 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:16:01.639799 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:16:01.640622 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:16:01.640629 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:16:01.640885 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:01.641594 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:16:01.641634 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:16:01.644794 (+ 3160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:16:01.647954 (+ 3160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:16:01.647966 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:16:01.647989 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:16:01.648616 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:16:01.648980 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:16:01.648987 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:16:01.648993 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:16:01.649275 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:16:01.649881 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:16:01.649892 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:16:01.650375 (+ 483us) client_negotiation.cc:769] Sending connection context
0924 16:16:01.650743 (+ 368us) client_negotiation.cc:240] Negotiation successful
0924 16:16:01.651366 (+ 623us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":331,"threads_started":1}
I20260924 16:16:01.654810 14317 heartbeater.cc:347] Connected to a master server at 127.29.234.62:44387
I20260924 16:16:01.655128 14317 heartbeater.cc:464] Registering TS with master...
I20260924 16:16:01.655894 14317 heartbeater.cc:511] Master 127.29.234.62:44387 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.29.234.3@KRBTEST.COM; defaulting to no policy
I20260924 16:16:01.657505 13987 ts_manager.cc:194] Registered new tserver with Master: ad7a4fa907754ea2849edb3d6089728f (127.29.234.2:37037)
I20260924 16:16:01.658340 13987 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.234.2@KRBTEST.COM'} at 127.29.234.2:52301
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab.
Entry for principal kudu/127.29.234.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab.
I20260924 16:16:01.689828 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.3:0
--local_ip_for_outbound_sockets=127.29.234.3
--webserver_interface=127.29.234.3
--webserver_port=0
--tserver_master_addrs=127.29.234.62:44387
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab
--principal=kudu/127.29.234.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:45871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:16:02.023900 14323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:16:02.024238 14323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:16:02.024407 14323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:16:02.031975 14323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:16:02.032130 14323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:16:02.032303 14323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.3
I20260924 16:16:02.042385 14323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:45871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab
--principal=kudu/127.29.234.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.234.3
--webserver_port=0
--tserver_master_addrs=127.29.234.62:44387
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.3
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:16:02.043818 14323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:16:02.045183 14323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:16:02.061686 14328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:16:02.061705 14329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:16:02.061734 14331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:16:02.062630 14323 server_base.cc:1034] running on GCE node
I20260924 16:16:02.063292 14323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:16:02.064338 14323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:16:02.065567 14323 hybrid_clock.cc:648] HybridClock initialized: now 1790266562065530 us; error 26 us; skew 500 ppm
I20260924 16:16:02.065899 14323 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:16:02 dist-test-slave-v5j9 krb5kdc[13940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266562, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:16:02.069003 14323 init.cc:377] Logged in from keytab as kudu/127.29.234.3@KRBTEST.COM (short username kudu)
I20260924 16:16:02.070713 14323 webserver.cc:460] Webserver started at http://127.29.234.3:43237/ using document root <none> and password file <none>
I20260924 16:16:02.071395 14323 fs_manager.cc:359] Metadata directory not provided
I20260924 16:16:02.071519 14323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:16:02.071837 14323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:16:02.074802 14323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-2/data/instance:
uuid: "0bef0b8e4f654d14bff64104add0e48b"
format_stamp: "Formatted at 2026-09-24 16:16:02 on dist-test-slave-v5j9"
server_key: "74d76fcce6e7c948021a25b67690f66b"
server_key_iv: "62210afdaef59af2c17ccd22fc60e4a2"
server_key_version: "encryptionkey@0"
I20260924 16:16:02.075512 14323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-2/wal/instance:
uuid: "0bef0b8e4f654d14bff64104add0e48b"
format_stamp: "Formatted at 2026-09-24 16:16:02 on dist-test-slave-v5j9"
server_key: "74d76fcce6e7c948021a25b67690f66b"
server_key_iv: "62210afdaef59af2c17ccd22fc60e4a2"
server_key_version: "encryptionkey@0"
I20260924 16:16:02.080554 14323 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260924 16:16:02.083884 14338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:16:02.085186 14323 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 16:16:02.085323 14323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-2/wal
uuid: "0bef0b8e4f654d14bff64104add0e48b"
format_stamp: "Formatted at 2026-09-24 16:16:02 on dist-test-slave-v5j9"
server_key: "74d76fcce6e7c948021a25b67690f66b"
server_key_iv: "62210afdaef59af2c17ccd22fc60e4a2"
server_key_version: "encryptionkey@0"
I20260924 16:16:02.085534 14323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:16:02.117540 14323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:16:02.121810 14323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:16:02.122051 14323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:16:02.122969 14323 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:16:02.124344 14323 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:16:02.124454 14323 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:16:02.124617 14323 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:16:02.124667 14323 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:16:02.136034 14181 heartbeater.cc:503] Master 127.29.234.62:44387 was elected leader, sending a full tablet report...
I20260924 16:16:02.157986 14323 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.3:40389
I20260924 16:16:02.158028 14452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.3:40389 every 8 connection(s)
I20260924 16:16:02.159381 14323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
I20260924 16:16:02.164143 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 14323
I20260924 16:16:02.164352 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-2/wal/instance
I20260924 16:16:02.165014 30632 external_mini_cluster.cc:1428] Setting key 5efd45e6cccde36228300f9c5cbadc41
Sep 24 16:16:02 dist-test-slave-v5j9 krb5kdc[13940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266562, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.3@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:16:02.181458 14455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:02.162077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.3:41187 (local address 127.29.234.62:44387)
0924 16:16:02.162548 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:02.162553 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:02.163454 (+ 901us) server_negotiation.cc:407] Connection header received
0924 16:16:02.166442 (+ 2988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:02.166449 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:02.166560 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:02.166674 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:02.168992 (+ 2318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:02.169871 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:02.171482 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:02.172005 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:02.172100 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:02.175384 (+ 3284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:16:02.175389 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:16:02.175392 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:16:02.175426 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:16:02.178165 (+ 2739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:02.178878 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:02.178881 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:02.178884 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:02.179019 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:02.179489 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:02.179493 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:02.179495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:02.179772 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:16:02.179949 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:02.180795 (+ 846us) server_negotiation.cc:299] Negotiation successful
0924 16:16:02.181024 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":249,"threads_started":1}
I20260924 16:16:02.182482 14456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:02.162613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:44387 (local address 127.29.234.3:41187)
0924 16:16:02.163295 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:02.163340 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 16:16:02.165821 (+ 2481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:16:02.167144 (+ 1323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:16:02.167158 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:16:02.167956 (+ 798us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:16:02.168849 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:16:02.168871 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:16:02.170004 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:16:02.170008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:16:02.171358 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:16:02.171370 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:16:02.172196 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:16:02.172202 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:16:02.172398 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:02.173008 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:16:02.173030 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:16:02.175179 (+ 2149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:16:02.178317 (+ 3138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:16:02.178326 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:16:02.178340 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:16:02.178764 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:16:02.179140 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:16:02.179145 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:16:02.179149 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:16:02.179350 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:16:02.179923 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:16:02.179932 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:16:02.180407 (+ 475us) client_negotiation.cc:769] Sending connection context
0924 16:16:02.180694 (+ 287us) client_negotiation.cc:240] Negotiation successful
0924 16:16:02.181252 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":280,"threads_started":1}
I20260924 16:16:02.183849 14453 heartbeater.cc:347] Connected to a master server at 127.29.234.62:44387
I20260924 16:16:02.184186 14453 heartbeater.cc:464] Registering TS with master...
I20260924 16:16:02.184962 14453 heartbeater.cc:511] Master 127.29.234.62:44387 requested a full tablet report, sending...
I20260924 16:16:02.186574 13987 ts_manager.cc:194] Registered new tserver with Master: 0bef0b8e4f654d14bff64104add0e48b (127.29.234.3:40389)
I20260924 16:16:02.187415 13987 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.234.3@KRBTEST.COM'} at 127.29.234.3:41187
I20260924 16:16:02.193621 30632 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:16:02.210912 14455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:02.198213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54928 (local address 127.29.234.62:44387)
0924 16:16:02.198470 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:02.198481 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:02.198836 (+ 355us) server_negotiation.cc:407] Connection header received
0924 16:16:02.198966 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:02.198970 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:02.199048 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:02.199130 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:02.200572 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:02.201405 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:02.202767 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:02.203176 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:02.203283 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:02.205901 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:16:02.205906 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:16:02.205910 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:16:02.205940 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:16:02.208453 (+ 2513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:02.209102 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:02.209106 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:02.209108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:02.209248 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:02.209636 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:02.209639 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:02.209641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:02.209942 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:16:02.210124 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:02.210430 (+ 306us) server_negotiation.cc:299] Negotiation successful
0924 16:16:02.210603 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260924 16:16:02.215005 13987 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54928:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:16:02.221922 13987 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:16:02.258209 14466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:02.234216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:34979 (local address 127.0.0.1:58766)
0924 16:16:02.246400 (+ 12184us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:02.246468 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0924 16:16:02.246716 (+ 248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:16:02.250725 (+ 4009us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:16:02.250730 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:16:02.250757 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:16:02.251283 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:16:02.251299 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:16:02.253690 (+ 2391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:16:02.253699 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:16:02.255689 (+ 1990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:16:02.255700 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:16:02.257139 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:16:02.257147 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:16:02.257319 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:02.257425 (+ 106us) client_negotiation.cc:769] Sending connection context
0924 16:16:02.257541 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 16:16:02.257766 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11775,"mutex_wait_us":64,"thread_start_us":12730,"threads_started":1}
I20260924 16:16:02.258234 14471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:02.237483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43240 (local address 127.29.234.3:40389)
0924 16:16:02.248720 (+ 11237us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:02.248729 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:02.250140 (+ 1411us) server_negotiation.cc:407] Connection header received
0924 16:16:02.250265 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:02.250274 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:02.250500 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:02.250704 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:16:02.251503 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:02.252954 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:02.255321 (+ 2367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:02.256823 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:02.256965 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:02.257215 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:02.257727 (+ 512us) server_negotiation.cc:299] Negotiation successful
0924 16:16:02.257908 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11043,"thread_start_us":11110,"threads_started":1}
I20260924 16:16:02.258234 14468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:02.234503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58766 (local address 127.29.234.1:34979)
0924 16:16:02.250097 (+ 15594us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:02.250107 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:02.250140 (+ 33us) server_negotiation.cc:407] Connection header received
0924 16:16:02.250239 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:02.250246 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:02.250497 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:02.251513 (+ 1016us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:16:02.251774 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:02.253519 (+ 1745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:02.255843 (+ 2324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:02.256881 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:02.257423 (+ 542us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:02.257683 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:02.257741 (+ 58us) server_negotiation.cc:299] Negotiation successful
0924 16:16:02.257896 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15360,"thread_start_us":15188,"threads_started":1}
I20260924 16:16:02.258945 14467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:02.236776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.3:40389 (local address 127.0.0.1:43240)
0924 16:16:02.249315 (+ 12539us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:02.249332 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:16:02.249483 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:16:02.250680 (+ 1197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:16:02.250685 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:16:02.250713 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:16:02.251291 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:16:02.251303 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:16:02.253125 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:16:02.253131 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:16:02.255079 (+ 1948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:16:02.255107 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:16:02.257037 (+ 1930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:16:02.257044 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:16:02.257288 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:02.257394 (+ 106us) client_negotiation.cc:769] Sending connection context
0924 16:16:02.257549 (+ 155us) client_negotiation.cc:240] Negotiation successful
0924 16:16:02.258532 (+ 983us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12420,"mutex_wait_us":815,"thread_start_us":11608,"threads_started":1}
I20260924 16:16:02.259248 14470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:02.234258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.2:37037 (local address 127.0.0.1:59544)
0924 16:16:02.248645 (+ 14387us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:02.248662 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:16:02.248802 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:16:02.249742 (+ 940us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:16:02.249751 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:16:02.249782 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:16:02.250200 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:16:02.250214 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:16:02.252435 (+ 2221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:16:02.252440 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:16:02.255088 (+ 2648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:16:02.255105 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:16:02.258329 (+ 3224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:16:02.258337 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:16:02.258531 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:02.258564 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 16:16:02.258649 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 16:16:02.258785 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14269,"thread_start_us":14811,"threads_started":1}
I20260924 16:16:02.259846 14469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:02.234602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59544 (local address 127.29.234.2:37037)
0924 16:16:02.248177 (+ 13575us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:02.248185 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:02.248870 (+ 685us) server_negotiation.cc:407] Connection header received
0924 16:16:02.248989 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:02.248999 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:02.249306 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:02.249534 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:16:02.250521 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:02.252261 (+ 1740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:02.255280 (+ 3019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:02.256798 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:02.257975 (+ 1177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:02.258171 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:02.258913 (+ 742us) server_negotiation.cc:299] Negotiation successful
0924 16:16:02.259540 (+ 627us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13412,"thread_start_us":13275,"threads_started":1}
I20260924 16:16:02.262606 14115 tablet_service.cc:1467] Processing CreateTablet for tablet 030ae721862e48df8e7a6a2a69d83737 (DEFAULT_TABLE table=test-table [id=4df5e1b6780540bfad0afc72f8f396f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:16:02.262845 14387 tablet_service.cc:1467] Processing CreateTablet for tablet 030ae721862e48df8e7a6a2a69d83737 (DEFAULT_TABLE table=test-table [id=4df5e1b6780540bfad0afc72f8f396f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:16:02.263193 14251 tablet_service.cc:1467] Processing CreateTablet for tablet 030ae721862e48df8e7a6a2a69d83737 (DEFAULT_TABLE table=test-table [id=4df5e1b6780540bfad0afc72f8f396f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:16:02.264430 14387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 030ae721862e48df8e7a6a2a69d83737. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:16:02.264446 14251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 030ae721862e48df8e7a6a2a69d83737. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:16:02.264454 14115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 030ae721862e48df8e7a6a2a69d83737. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:16:02.277254 14472 tablet_bootstrap.cc:492] T 030ae721862e48df8e7a6a2a69d83737 P 4287e1c923de4b69a9c02d00cf413ae7: Bootstrap starting.
I20260924 16:16:02.279549 14473 tablet_bootstrap.cc:492] T 030ae721862e48df8e7a6a2a69d83737 P 0bef0b8e4f654d14bff64104add0e48b: Bootstrap starting.
I20260924 16:16:02.281332 14472 tablet_bootstrap.cc:654] T 030ae721862e48df8e7a6a2a69d83737 P 4287e1c923de4b69a9c02d00cf413ae7: Neither blocks nor log segments found. Creating new log.
I20260924 16:16:02.281447 14474 tablet_bootstrap.cc:492] T 030ae721862e48df8e7a6a2a69d83737 P ad7a4fa907754ea2849edb3d6089728f: Bootstrap starting.
I20260924 16:16:02.283193 14473 tablet_bootstrap.cc:654] T 030ae721862e48df8e7a6a2a69d83737 P 0bef0b8e4f654d14bff64104add0e48b: Neither blocks nor log segments found. Creating new log.
I20260924 16:16:02.283416 14472 log.cc:824] T 030ae721862e48df8e7a6a2a69d83737 P 4287e1c923de4b69a9c02d00cf413ae7: Log is configured to *not* fsync() on all Append() calls
I20260924 16:16:02.284430 14473 log.cc:824] T 030ae721862e48df8e7a6a2a69d83737 P 0bef0b8e4f654d14bff64104add0e48b: Log is configured to *not* fsync() on all Append() calls
I20260924 16:16:02.284850 14474 tablet_bootstrap.cc:654] T 030ae721862e48df8e7a6a2a69d83737 P ad7a4fa907754ea2849edb3d6089728f: Neither blocks nor log segments found. Creating new log.
I20260924 16:16:02.286154 14474 log.cc:824] T 030ae721862e48df8e7a6a2a69d83737 P ad7a4fa907754ea2849edb3d6089728f: Log is configured to *not* fsync() on all Append() calls
I20260924 16:16:02.286847 14472 tablet_bootstrap.cc:492] T 030ae721862e48df8e7a6a2a69d83737 P 4287e1c923de4b69a9c02d00cf413ae7: No bootstrap required, opened a new log
I20260924 16:16:02.287137 14472 ts_tablet_manager.cc:1397] T 030ae721862e48df8e7a6a2a69d83737 P 4287e1c923de4b69a9c02d00cf413ae7: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260924 16:16:02.288157 14473 tablet_bootstrap.cc:492] T 030ae721862e48df8e7a6a2a69d83737 P 0bef0b8e4f654d14bff64104add0e48b: No bootstrap required, opened a new log
I20260924 16:16:02.288440 14473 ts_tablet_manager.cc:1397] T 030ae721862e48df8e7a6a2a69d83737 P 0bef0b8e4f654d14bff64104add0e48b: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260924 16:16:02.288874 14474 tablet_bootstrap.cc:492] T 030ae721862e48df8e7a6a2a69d83737 P ad7a4fa907754ea2849edb3d6089728f: No bootstrap required, opened a new log
I20260924 16:16:02.289143 14474 ts_tablet_manager.cc:1397] T 030ae721862e48df8e7a6a2a69d83737 P ad7a4fa907754ea2849edb3d6089728f: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260924 16:16:02.297284 14474 raft_consensus.cc:348] T 030ae721862e48df8e7a6a2a69d83737 P ad7a4fa907754ea2849edb3d6089728f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4287e1c923de4b69a9c02d00cf413ae7" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 34979 } } peers { permanent_uuid: "ad7a4fa907754ea2849edb3d6089728f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 37037 } } peers { permanent_uuid: "0bef0b8e4f654d14bff64104add0e48b" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 40389 } }
I20260924 16:16:02.297791 14474 raft_consensus.cc:374] T 030ae721862e48df8e7a6a2a69d83737 P ad7a4fa907754ea2849edb3d6089728f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:16:02.297896 14474 raft_consensus.cc:729] T 030ae721862e48df8e7a6a2a69d83737 P ad7a4fa907754ea2849edb3d6089728f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad7a4fa907754ea2849edb3d6089728f, State: Initialized, Role: FOLLOWER
I20260924 16:16:02.298125 14472 raft_consensus.cc:348] T 030ae721862e48df8e7a6a2a69d83737 P 4287e1c923de4b69a9c02d00cf413ae7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4287e1c923de4b69a9c02d00cf413ae7" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 34979 } } peers { permanent_uuid: "ad7a4fa907754ea2849edb3d6089728f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 37037 } } peers { permanent_uuid: "0bef0b8e4f654d14bff64104add0e48b" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 40389 } }
I20260924 16:16:02.298563 14472 raft_consensus.cc:374] T 030ae721862e48df8e7a6a2a69d83737 P 4287e1c923de4b69a9c02d00cf413ae7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:16:02.298666 14472 raft_consensus.cc:729] T 030ae721862e48df8e7a6a2a69d83737 P 4287e1c923de4b69a9c02d00cf413ae7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4287e1c923de4b69a9c02d00cf413ae7, State: Initialized, Role: FOLLOWER
I20260924 16:16:02.298597 14474 consensus_queue.cc:260] T 030ae721862e48df8e7a6a2a69d83737 P ad7a4fa907754ea2849edb3d6089728f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4287e1c923de4b69a9c02d00cf413ae7" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 34979 } } peers { permanent_uuid: "ad7a4fa907754ea2849edb3d6089728f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 37037 } } peers { permanent_uuid: "0bef0b8e4f654d14bff64104add0e48b" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 40389 } }
I20260924 16:16:02.299211 14473 raft_consensus.cc:348] T 030ae721862e48df8e7a6a2a69d83737 P 0bef0b8e4f654d14bff64104add0e48b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4287e1c923de4b69a9c02d00cf413ae7" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 34979 } } peers { permanent_uuid: "ad7a4fa907754ea2849edb3d6089728f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 37037 } } peers { permanent_uuid: "0bef0b8e4f654d14bff64104add0e48b" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 40389 } }
I20260924 16:16:02.299400 14472 consensus_queue.cc:260] T 030ae721862e48df8e7a6a2a69d83737 P 4287e1c923de4b69a9c02d00cf413ae7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4287e1c923de4b69a9c02d00cf413ae7" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 34979 } } peers { permanent_uuid: "ad7a4fa907754ea2849edb3d6089728f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 37037 } } peers { permanent_uuid: "0bef0b8e4f654d14bff64104add0e48b" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 40389 } }
I20260924 16:16:02.299633 14473 raft_consensus.cc:374] T 030ae721862e48df8e7a6a2a69d83737 P 0bef0b8e4f654d14bff64104add0e48b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:16:02.299737 14473 raft_consensus.cc:729] T 030ae721862e48df8e7a6a2a69d83737 P 0bef0b8e4f654d14bff64104add0e48b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bef0b8e4f654d14bff64104add0e48b, State: Initialized, Role: FOLLOWER
I20260924 16:16:02.300463 14473 consensus_queue.cc:260] T 030ae721862e48df8e7a6a2a69d83737 P 0bef0b8e4f654d14bff64104add0e48b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4287e1c923de4b69a9c02d00cf413ae7" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 34979 } } peers { permanent_uuid: "ad7a4fa907754ea2849edb3d6089728f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 37037 } } peers { permanent_uuid: "0bef0b8e4f654d14bff64104add0e48b" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 40389 } }
I20260924 16:16:02.302824 14473 ts_tablet_manager.cc:1428] T 030ae721862e48df8e7a6a2a69d83737 P 0bef0b8e4f654d14bff64104add0e48b: Time spent starting tablet: real 0.014s user 0.013s sys 0.002s
I20260924 16:16:02.303373 14453 heartbeater.cc:503] Master 127.29.234.62:44387 was elected leader, sending a full tablet report...
I20260924 16:16:02.304564 14472 ts_tablet_manager.cc:1428] T 030ae721862e48df8e7a6a2a69d83737 P 4287e1c923de4b69a9c02d00cf413ae7: Time spent starting tablet: real 0.017s user 0.010s sys 0.007s
I20260924 16:16:02.305649 14317 heartbeater.cc:503] Master 127.29.234.62:44387 was elected leader, sending a full tablet report...
I20260924 16:16:02.306372 14474 ts_tablet_manager.cc:1428] T 030ae721862e48df8e7a6a2a69d83737 P ad7a4fa907754ea2849edb3d6089728f: Time spent starting tablet: real 0.017s user 0.018s sys 0.000s
W20260924 16:16:02.353742 14182 tablet.cc:2369] T 030ae721862e48df8e7a6a2a69d83737 P 4287e1c923de4b69a9c02d00cf413ae7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 16:16:02.381618 14318 tablet.cc:2369] T 030ae721862e48df8e7a6a2a69d83737 P ad7a4fa907754ea2849edb3d6089728f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:16:02.405159 14478 raft_consensus.cc:482] T 030ae721862e48df8e7a6a2a69d83737 P ad7a4fa907754ea2849edb3d6089728f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:16:02.405550 14478 raft_consensus.cc:504] T 030ae721862e48df8e7a6a2a69d83737 P ad7a4fa907754ea2849edb3d6089728f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4287e1c923de4b69a9c02d00cf413ae7" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 34979 } } peers { permanent_uuid: "ad7a4fa907754ea2849edb3d6089728f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 37037 } } peers { permanent_uuid: "0bef0b8e4f654d14bff64104add0e48b" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 40389 } }
I20260924 16:16:02.407423 14478 leader_election.cc:288] T 030ae721862e48df8e7a6a2a69d83737 P ad7a4fa907754ea2849edb3d6089728f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4287e1c923de4b69a9c02d00cf413ae7 (127.29.234.1:34979), 0bef0b8e4f654d14bff64104add0e48b (127.29.234.3:40389)
W20260924 16:16:02.411120 14454 tablet.cc:2369] T 030ae721862e48df8e7a6a2a69d83737 P 0bef0b8e4f654d14bff64104add0e48b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:16:02.414948 14471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:02.407913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.2:39321 (local address 127.29.234.3:40389)
0924 16:16:02.408136 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:02.408144 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:02.409328 (+ 1184us) server_negotiation.cc:407] Connection header received
0924 16:16:02.409544 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:02.409549 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:02.409614 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:02.409713 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:16:02.410299 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:02.411109 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:02.413178 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:02.414007 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:02.414207 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:02.414282 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:02.414561 (+ 279us) server_negotiation.cc:299] Negotiation successful
0924 16:16:02.414695 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 16:16:02.415021 14482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:02.408294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.3:40389 (local address 127.29.234.2:39321)
0924 16:16:02.409217 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:02.409231 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:16:02.409401 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:16:02.409734 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:16:02.409742 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:16:02.409767 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:16:02.410140 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:16:02.410149 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:16:02.411272 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:16:02.411277 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:16:02.413026 (+ 1749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:16:02.413038 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:16:02.414235 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:16:02.414241 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:16:02.414406 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:02.414428 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 16:16:02.414536 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 16:16:02.414657 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":780,"thread_start_us":436,"threads_started":1}
I20260924 16:16:02.415031 14468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:02.407914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.2:39623 (local address 127.29.234.1:34979)
0924 16:16:02.408126 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:02.408132 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:02.409067 (+ 935us) server_negotiation.cc:407] Connection header received
0924 16:16:02.409296 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:02.409300 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:02.409417 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:02.409511 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:16:02.410065 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:02.410856 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:02.413067 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:02.414122 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:02.414295 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:02.414375 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:02.414655 (+ 280us) server_negotiation.cc:299] Negotiation successful
0924 16:16:02.414783 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 16:16:02.415184 14481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:02.408165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:34979 (local address 127.29.234.2:39623)
0924 16:16:02.408947 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:02.408969 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 16:16:02.409141 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:16:02.409526 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:16:02.409531 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:16:02.409554 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:16:02.409926 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:16:02.409934 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:16:02.410998 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:16:02.411004 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:16:02.412886 (+ 1882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:16:02.412898 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:16:02.414350 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:16:02.414356 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:16:02.414492 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:02.414514 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 16:16:02.414620 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 16:16:02.414739 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"mutex_wait_us":38,"thread_start_us":429,"threads_started":1}
I20260924 16:16:02.416247 14407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "030ae721862e48df8e7a6a2a69d83737" candidate_uuid: "ad7a4fa907754ea2849edb3d6089728f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bef0b8e4f654d14bff64104add0e48b" is_pre_election: true
I20260924 16:16:02.416249 14136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "030ae721862e48df8e7a6a2a69d83737" candidate_uuid: "ad7a4fa907754ea2849edb3d6089728f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4287e1c923de4b69a9c02d00cf413ae7" is_pre_election: true
I20260924 16:16:02.416826 14136 raft_consensus.cc:2454] T 030ae721862e48df8e7a6a2a69d83737 P 4287e1c923de4b69a9c02d00cf413ae7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad7a4fa907754ea2849edb3d6089728f in term 0.
I20260924 16:16:02.416826 14407 raft_consensus.cc:2454] T 030ae721862e48df8e7a6a2a69d83737 P 0bef0b8e4f654d14bff64104add0e48b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad7a4fa907754ea2849edb3d6089728f in term 0.
I20260924 16:16:02.417629 14205 leader_election.cc:302] T 030ae721862e48df8e7a6a2a69d83737 P ad7a4fa907754ea2849edb3d6089728f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bef0b8e4f654d14bff64104add0e48b, ad7a4fa907754ea2849edb3d6089728f; no voters:
I20260924 16:16:02.418056 14478 raft_consensus.cc:2781] T 030ae721862e48df8e7a6a2a69d83737 P ad7a4fa907754ea2849edb3d6089728f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:16:02.418202 14478 raft_consensus.cc:482] T 030ae721862e48df8e7a6a2a69d83737 P ad7a4fa907754ea2849edb3d6089728f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:16:02.418311 14478 raft_consensus.cc:3037] T 030ae721862e48df8e7a6a2a69d83737 P ad7a4fa907754ea2849edb3d6089728f [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:16:02.420562 14478 raft_consensus.cc:504] T 030ae721862e48df8e7a6a2a69d83737 P ad7a4fa907754ea2849edb3d6089728f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4287e1c923de4b69a9c02d00cf413ae7" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 34979 } } peers { permanent_uuid: "ad7a4fa907754ea2849edb3d6089728f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 37037 } } peers { permanent_uuid: "0bef0b8e4f654d14bff64104add0e48b" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 40389 } }
I20260924 16:16:02.421173 14478 leader_election.cc:288] T 030ae721862e48df8e7a6a2a69d83737 P ad7a4fa907754ea2849edb3d6089728f [CANDIDATE]: Term 1 election: Requested vote from peers 4287e1c923de4b69a9c02d00cf413ae7 (127.29.234.1:34979), 0bef0b8e4f654d14bff64104add0e48b (127.29.234.3:40389)
I20260924 16:16:02.421661 14136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "030ae721862e48df8e7a6a2a69d83737" candidate_uuid: "ad7a4fa907754ea2849edb3d6089728f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4287e1c923de4b69a9c02d00cf413ae7"
I20260924 16:16:02.421661 14407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "030ae721862e48df8e7a6a2a69d83737" candidate_uuid: "ad7a4fa907754ea2849edb3d6089728f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bef0b8e4f654d14bff64104add0e48b"
I20260924 16:16:02.421878 14407 raft_consensus.cc:3037] T 030ae721862e48df8e7a6a2a69d83737 P 0bef0b8e4f654d14bff64104add0e48b [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:16:02.421878 14136 raft_consensus.cc:3037] T 030ae721862e48df8e7a6a2a69d83737 P 4287e1c923de4b69a9c02d00cf413ae7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:16:02.425083 14136 raft_consensus.cc:2454] T 030ae721862e48df8e7a6a2a69d83737 P 4287e1c923de4b69a9c02d00cf413ae7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad7a4fa907754ea2849edb3d6089728f in term 1.
I20260924 16:16:02.425083 14407 raft_consensus.cc:2454] T 030ae721862e48df8e7a6a2a69d83737 P 0bef0b8e4f654d14bff64104add0e48b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad7a4fa907754ea2849edb3d6089728f in term 1.
I20260924 16:16:02.425522 14206 leader_election.cc:302] T 030ae721862e48df8e7a6a2a69d83737 P ad7a4fa907754ea2849edb3d6089728f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4287e1c923de4b69a9c02d00cf413ae7, ad7a4fa907754ea2849edb3d6089728f; no voters:
I20260924 16:16:02.425793 14478 raft_consensus.cc:2781] T 030ae721862e48df8e7a6a2a69d83737 P ad7a4fa907754ea2849edb3d6089728f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:16:02.426234 14478 raft_consensus.cc:686] T 030ae721862e48df8e7a6a2a69d83737 P ad7a4fa907754ea2849edb3d6089728f [term 1 LEADER]: Becoming Leader. State: Replica: ad7a4fa907754ea2849edb3d6089728f, State: Running, Role: LEADER
I20260924 16:16:02.426723 14478 consensus_queue.cc:237] T 030ae721862e48df8e7a6a2a69d83737 P ad7a4fa907754ea2849edb3d6089728f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4287e1c923de4b69a9c02d00cf413ae7" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 34979 } } peers { permanent_uuid: "ad7a4fa907754ea2849edb3d6089728f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 37037 } } peers { permanent_uuid: "0bef0b8e4f654d14bff64104add0e48b" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 40389 } }
I20260924 16:16:02.431545 13987 catalog_manager.cc:5495] T 030ae721862e48df8e7a6a2a69d83737 P ad7a4fa907754ea2849edb3d6089728f reported cstate change: term changed from 0 to 1, leader changed from <none> to ad7a4fa907754ea2849edb3d6089728f (127.29.234.2). New cstate: current_term: 1 leader_uuid: "ad7a4fa907754ea2849edb3d6089728f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4287e1c923de4b69a9c02d00cf413ae7" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 34979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad7a4fa907754ea2849edb3d6089728f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 37037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0bef0b8e4f654d14bff64104add0e48b" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 40389 } health_report { overall_health: UNKNOWN } } }
I20260924 16:16:02.455953 14469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:02.450092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42632 (local address 127.29.234.2:37037)
0924 16:16:02.450330 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:02.450337 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:02.450357 (+ 20us) server_negotiation.cc:407] Connection header received
0924 16:16:02.450598 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:02.450656 (+ 58us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:02.450773 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:02.450940 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:16:02.451371 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:02.452091 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:02.453746 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:02.454168 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:02.454304 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:02.454814 (+ 510us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:16:02.455359 (+ 545us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:16:02.455523 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:02.455581 (+ 58us) server_negotiation.cc:299] Negotiation successful
0924 16:16:02.455673 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 16:16:02.464377 14407 raft_consensus.cc:1261] T 030ae721862e48df8e7a6a2a69d83737 P 0bef0b8e4f654d14bff64104add0e48b [term 1 FOLLOWER]: Refusing update from remote peer ad7a4fa907754ea2849edb3d6089728f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:16:02.464377 14136 raft_consensus.cc:1261] T 030ae721862e48df8e7a6a2a69d83737 P 4287e1c923de4b69a9c02d00cf413ae7 [term 1 FOLLOWER]: Refusing update from remote peer ad7a4fa907754ea2849edb3d6089728f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:16:02.465516 14483 consensus_queue.cc:1035] T 030ae721862e48df8e7a6a2a69d83737 P ad7a4fa907754ea2849edb3d6089728f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bef0b8e4f654d14bff64104add0e48b" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 40389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:16:02.465842 14478 consensus_queue.cc:1035] T 030ae721862e48df8e7a6a2a69d83737 P ad7a4fa907754ea2849edb3d6089728f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4287e1c923de4b69a9c02d00cf413ae7" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 34979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:16:02.477291 14486 mvcc.cc:205] Tried to move back new op lower bound from 7332931839843442688 to 7332931839703785472. Current Snapshot: MvccSnapshot[applied={T|T < 7332931839843442688}]
I20260924 16:16:02.493513 14487 mvcc.cc:205] Tried to move back new op lower bound from 7332931839843442688 to 7332931839703785472. Current Snapshot: MvccSnapshot[applied={T|T < 7332931839843442688}]
I20260924 16:16:02.503104 13987 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54928:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:16:02.503419 13987 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54928:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:16:02.507364 13987 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 92a389a3e45944bf836899bbf5e85366: Sending DeleteTablet for 3 replicas of tablet 030ae721862e48df8e7a6a2a69d83737
I20260924 16:16:02.508517 14115 tablet_service.cc:1514] Processing DeleteTablet for tablet 030ae721862e48df8e7a6a2a69d83737 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:16:02 UTC) from {username='kudu', principal='kudu/127.29.234.62@KRBTEST.COM'} at 127.0.0.1:58766
I20260924 16:16:02.508649 14251 tablet_service.cc:1514] Processing DeleteTablet for tablet 030ae721862e48df8e7a6a2a69d83737 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:16:02 UTC) from {username='kudu', principal='kudu/127.29.234.62@KRBTEST.COM'} at 127.0.0.1:59544
I20260924 16:16:02.508796 14387 tablet_service.cc:1514] Processing DeleteTablet for tablet 030ae721862e48df8e7a6a2a69d83737 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:16:02 UTC) from {username='kudu', principal='kudu/127.29.234.62@KRBTEST.COM'} at 127.0.0.1:43240
I20260924 16:16:02.509706 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 14050
I20260924 16:16:02.512329 14497 tablet_replica.cc:330] T 030ae721862e48df8e7a6a2a69d83737 P ad7a4fa907754ea2849edb3d6089728f: stopping tablet replica
I20260924 16:16:02.512871 14497 raft_consensus.cc:2229] T 030ae721862e48df8e7a6a2a69d83737 P ad7a4fa907754ea2849edb3d6089728f [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:16:02.513621 14497 raft_consensus.cc:2258] T 030ae721862e48df8e7a6a2a69d83737 P ad7a4fa907754ea2849edb3d6089728f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:16:02.514470 14498 tablet_replica.cc:330] T 030ae721862e48df8e7a6a2a69d83737 P 0bef0b8e4f654d14bff64104add0e48b: stopping tablet replica
I20260924 16:16:02.514935 14498 raft_consensus.cc:2229] T 030ae721862e48df8e7a6a2a69d83737 P 0bef0b8e4f654d14bff64104add0e48b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:16:02.515486 14498 raft_consensus.cc:2258] T 030ae721862e48df8e7a6a2a69d83737 P 0bef0b8e4f654d14bff64104add0e48b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:16:02.517411 14496 tablet_replica.cc:330] T 030ae721862e48df8e7a6a2a69d83737 P 4287e1c923de4b69a9c02d00cf413ae7: stopping tablet replica
I20260924 16:16:02.517838 14496 raft_consensus.cc:2229] T 030ae721862e48df8e7a6a2a69d83737 P 4287e1c923de4b69a9c02d00cf413ae7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:16:02.518062 14498 ts_tablet_manager.cc:1905] T 030ae721862e48df8e7a6a2a69d83737 P 0bef0b8e4f654d14bff64104add0e48b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:16:02.518332 14496 raft_consensus.cc:2258] T 030ae721862e48df8e7a6a2a69d83737 P 4287e1c923de4b69a9c02d00cf413ae7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:16:02.521632 14497 ts_tablet_manager.cc:1905] T 030ae721862e48df8e7a6a2a69d83737 P ad7a4fa907754ea2849edb3d6089728f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:16:02.526170 14498 ts_tablet_manager.cc:1918] T 030ae721862e48df8e7a6a2a69d83737 P 0bef0b8e4f654d14bff64104add0e48b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:16:02.526407 14498 log.cc:1196] T 030ae721862e48df8e7a6a2a69d83737 P 0bef0b8e4f654d14bff64104add0e48b: Deleting WAL directory at /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-2/wal/wals/030ae721862e48df8e7a6a2a69d83737
I20260924 16:16:02.527040 14498 ts_tablet_manager.cc:1939] T 030ae721862e48df8e7a6a2a69d83737 P 0bef0b8e4f654d14bff64104add0e48b: Deleting consensus metadata
Sep 24 16:16:02 dist-test-slave-v5j9 krb5kdc[13940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.234.1@KRBTEST.COM
I20260924 16:16:02.529258 13971 catalog_manager.cc:4841] TS 0bef0b8e4f654d14bff64104add0e48b (127.29.234.3:40389): tablet 030ae721862e48df8e7a6a2a69d83737 (table test-table [id=4df5e1b6780540bfad0afc72f8f396f0]) successfully deleted
I20260924 16:16:02.529577 14497 ts_tablet_manager.cc:1918] T 030ae721862e48df8e7a6a2a69d83737 P ad7a4fa907754ea2849edb3d6089728f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:16:02.529789 14497 log.cc:1196] T 030ae721862e48df8e7a6a2a69d83737 P ad7a4fa907754ea2849edb3d6089728f: Deleting WAL directory at /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-1/wal/wals/030ae721862e48df8e7a6a2a69d83737
I20260924 16:16:02.530349 14497 ts_tablet_manager.cc:1939] T 030ae721862e48df8e7a6a2a69d83737 P ad7a4fa907754ea2849edb3d6089728f: Deleting consensus metadata
I20260924 16:16:02.532366 13973 catalog_manager.cc:4841] TS ad7a4fa907754ea2849edb3d6089728f (127.29.234.2:37037): tablet 030ae721862e48df8e7a6a2a69d83737 (table test-table [id=4df5e1b6780540bfad0afc72f8f396f0]) successfully deleted
I20260924 16:16:02.532430 14496 ts_tablet_manager.cc:1905] T 030ae721862e48df8e7a6a2a69d83737 P 4287e1c923de4b69a9c02d00cf413ae7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:16:02.537415 14468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:02.510741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52792 (local address 127.29.234.1:34979)
0924 16:16:02.510902 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:02.510908 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:02.516709 (+ 5801us) server_negotiation.cc:407] Connection header received
0924 16:16:02.516945 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:02.516955 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:02.517065 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:02.517202 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:02.519453 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:02.520589 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:02.522700 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:02.523435 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:02.523540 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:02.529454 (+ 5914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:16:02.529472 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:16:02.529493 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:16:02.529549 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:16:02.533612 (+ 4063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:02.534602 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:02.534613 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:02.534617 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:02.534822 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:02.535354 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:02.535360 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:02.535364 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:02.535866 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:16:02.536209 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:02.536608 (+ 399us) server_negotiation.cc:299] Negotiation successful
0924 16:16:02.536909 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 16:16:02.538555 14176 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:16:02.628252 14496 ts_tablet_manager.cc:1918] T 030ae721862e48df8e7a6a2a69d83737 P 4287e1c923de4b69a9c02d00cf413ae7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:16:02.628496 14496 log.cc:1196] T 030ae721862e48df8e7a6a2a69d83737 P 4287e1c923de4b69a9c02d00cf413ae7: Deleting WAL directory at /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-0/wal/wals/030ae721862e48df8e7a6a2a69d83737
I20260924 16:16:02.629073 14496 ts_tablet_manager.cc:1939] T 030ae721862e48df8e7a6a2a69d83737 P 4287e1c923de4b69a9c02d00cf413ae7: Deleting consensus metadata
I20260924 16:16:02.629290 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 14050
I20260924 16:16:02.648952 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 14187
W20260924 16:16:02.648952 13972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.234.1:34979 (error 108)
I20260924 16:16:02.655892 14470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:02.655440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:34979 (local address 127.0.0.1:52798)
0924 16:16:02.655582 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:02.655704 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.1:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260924 16:16:02.656244 13972 catalog_manager.cc:4584] TS 4287e1c923de4b69a9c02d00cf413ae7 (127.29.234.1:34979): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 030ae721862e48df8e7a6a2a69d83737: Network error: Client connection negotiation failed: client connection to 127.29.234.1:34979: connect: Connection refused (error 111)
Sep 24 16:16:02 dist-test-slave-v5j9 krb5kdc[13940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.234.2@KRBTEST.COM
I20260924 16:16:02.669021 14469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:02.650836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42636 (local address 127.29.234.2:37037)
0924 16:16:02.651070 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:02.651079 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:02.651101 (+ 22us) server_negotiation.cc:407] Connection header received
0924 16:16:02.651180 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:02.651185 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:02.651277 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:02.651408 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:02.654343 (+ 2935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:02.655594 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:02.657289 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:02.657811 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:02.657956 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:02.662209 (+ 4253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:16:02.662223 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:16:02.662244 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:16:02.662294 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:16:02.666231 (+ 3937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:02.666883 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:02.666893 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:02.666896 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:02.667062 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:02.667423 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:02.667429 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:02.667432 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:02.667892 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:16:02.668216 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:02.668405 (+ 189us) server_negotiation.cc:299] Negotiation successful
0924 16:16:02.668621 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 16:16:02.669842 14312 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:16:02.709978 14470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:02.709433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:34979 (local address 127.0.0.1:52810)
0924 16:16:02.709662 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:02.709780 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.1:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260924 16:16:02.711071 14470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:02.710672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:34979 (local address 127.0.0.1:52818)
0924 16:16:02.710837 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:02.710923 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.1:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260924 16:16:02.771857 14470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:02.771336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:34979 (local address 127.0.0.1:52824)
0924 16:16:02.771593 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:02.771694 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.1:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260924 16:16:02.773217 14470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:02.772637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:34979 (local address 127.0.0.1:52834)
0924 16:16:02.772804 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:02.773055 (+ 251us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.1:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260924 16:16:02.786460 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 14187
I20260924 16:16:02.806515 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 14323
Sep 24 16:16:02 dist-test-slave-v5j9 krb5kdc[13940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.234.3@KRBTEST.COM
I20260924 16:16:02.822158 14471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:02.807284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43824 (local address 127.29.234.3:40389)
0924 16:16:02.807466 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:02.807471 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:02.807608 (+ 137us) server_negotiation.cc:407] Connection header received
0924 16:16:02.807822 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:02.807826 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:02.807899 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:02.807981 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:02.809260 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:02.810065 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:02.811381 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:02.811767 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:02.811857 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:02.816521 (+ 4664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:16:02.816538 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:16:02.816558 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:16:02.816595 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:16:02.819461 (+ 2866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:02.820155 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:02.820162 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:02.820164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:02.820309 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:02.820689 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:02.820692 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:02.820694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:02.821105 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:16:02.821338 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:02.821738 (+ 400us) server_negotiation.cc:299] Negotiation successful
0924 16:16:02.821906 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 16:16:02.822862 14448 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:16:02.853895 14470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:02.853294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:34979 (local address 127.0.0.1:52844)
0924 16:16:02.853559 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:02.853680 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.1:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260924 16:16:02.855077 14470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:02.854669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:34979 (local address 127.0.0.1:52850)
0924 16:16:02.854862 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:02.854932 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.1:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260924 16:16:02.914081 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 14323
I20260924 16:16:02.931457 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 13953
I20260924 16:16:02.932734 14019 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:16:03.042583 14470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:03.041864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:34979 (local address 127.0.0.1:52866)
0924 16:16:03.042197 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:03.042347 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.1:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192}
I20260924 16:16:03.042709 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 13953
I20260924 16:16:23.065685 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:44387
--webserver_interface=127.29.234.62
--webserver_port=41217
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab
--principal=kudu/127.29.234.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:45871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.234.62:44387
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:16:23.411564 14505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:16:23.411868 14505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:16:23.412037 14505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:16:23.419365 14505 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 16:16:23.419459 14505 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 16:16:23.419546 14505 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 16:16:23.419624 14505 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:16:23.419718 14505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:16:23.419832 14505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:16:23.419910 14505 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:16:23.419992 14505 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:16:23.430112 14505 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:45871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.29.234.62:44387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab
--principal=kudu/127.29.234.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:44387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.234.62
--webserver_port=41217
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:16:23.431514 14505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:16:23.432934 14505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 16:16:23.456322 14505 server_base.cc:1034] running on GCE node
W20260924 16:16:23.456248 14512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:16:23.456348 14514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:16:23.456226 14511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:16:23.457684 14505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:16:23.459270 14505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:16:23.460528 14505 hybrid_clock.cc:648] HybridClock initialized: now 1790266583460462 us; error 33 us; skew 500 ppm
I20260924 16:16:23.460932 14505 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:16:23 dist-test-slave-v5j9 krb5kdc[13940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266583, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:16:23.464526 14505 init.cc:377] Logged in from keytab as kudu/127.29.234.62@KRBTEST.COM (short username kudu)
I20260924 16:16:23.466341 14505 webserver.cc:460] Webserver started at http://127.29.234.62:41217/ using document root <none> and password file <none>
I20260924 16:16:23.467208 14505 fs_manager.cc:359] Metadata directory not provided
I20260924 16:16:23.467370 14505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:16:23.474033 14505 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 16:16:23.477147 14521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:16:23.478647 14505 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:16:23.478845 14505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/master-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/master-0/wal
uuid: "92a389a3e45944bf836899bbf5e85366"
format_stamp: "Formatted at 2026-09-24 16:16:00 on dist-test-slave-v5j9"
server_key: "cc5bc5cfa98f40f77611023dcbd525df"
server_key_iv: "e93ec137cc70329d4350bc2009d92071"
server_key_version: "encryptionkey@0"
I20260924 16:16:23.479954 14505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:16:23.506397 14505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:16:23.512050 14505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:16:23.512434 14505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:16:23.533677 14505 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.62:44387
I20260924 16:16:23.533687 14573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.62:44387 every 8 connection(s)
I20260924 16:16:23.535451 14505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
I20260924 16:16:23.541718 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 14505
I20260924 16:16:23.542797 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.1:34979
--local_ip_for_outbound_sockets=127.29.234.1
--tserver_master_addrs=127.29.234.62:44387
--webserver_port=43803
--webserver_interface=127.29.234.1
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab
--principal=kudu/127.29.234.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:45871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:16:23.543889 14574 sys_catalog.cc:263] Verifying existing consensus state
I20260924 16:16:23.549604 14574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92a389a3e45944bf836899bbf5e85366: Bootstrap starting.
I20260924 16:16:23.571339 14574 log.cc:824] T 00000000000000000000000000000000 P 92a389a3e45944bf836899bbf5e85366: Log is configured to *not* fsync() on all Append() calls
I20260924 16:16:23.583983 14574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92a389a3e45944bf836899bbf5e85366: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 16:16:23.584573 14574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92a389a3e45944bf836899bbf5e85366: Bootstrap complete.
I20260924 16:16:23.594326 14574 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92a389a3e45944bf836899bbf5e85366 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92a389a3e45944bf836899bbf5e85366" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 44387 } }
I20260924 16:16:23.595194 14574 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92a389a3e45944bf836899bbf5e85366 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92a389a3e45944bf836899bbf5e85366, State: Initialized, Role: FOLLOWER
I20260924 16:16:23.595947 14574 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92a389a3e45944bf836899bbf5e85366 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92a389a3e45944bf836899bbf5e85366" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 44387 } }
I20260924 16:16:23.596201 14574 raft_consensus.cc:388] T 00000000000000000000000000000000 P 92a389a3e45944bf836899bbf5e85366 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:16:23.596323 14574 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92a389a3e45944bf836899bbf5e85366 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:16:23.596524 14574 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92a389a3e45944bf836899bbf5e85366 [term 1 FOLLOWER]: Advancing to term 2
I20260924 16:16:23.600431 14574 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92a389a3e45944bf836899bbf5e85366 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92a389a3e45944bf836899bbf5e85366" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 44387 } }
I20260924 16:16:23.600971 14574 leader_election.cc:302] T 00000000000000000000000000000000 P 92a389a3e45944bf836899bbf5e85366 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92a389a3e45944bf836899bbf5e85366; no voters:
I20260924 16:16:23.601692 14574 leader_election.cc:288] T 00000000000000000000000000000000 P 92a389a3e45944bf836899bbf5e85366 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 16:16:23.601830 14578 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92a389a3e45944bf836899bbf5e85366 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 16:16:23.603106 14578 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92a389a3e45944bf836899bbf5e85366 [term 2 LEADER]: Becoming Leader. State: Replica: 92a389a3e45944bf836899bbf5e85366, State: Running, Role: LEADER
I20260924 16:16:23.603797 14578 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92a389a3e45944bf836899bbf5e85366 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92a389a3e45944bf836899bbf5e85366" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 44387 } }
I20260924 16:16:23.604184 14574 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92a389a3e45944bf836899bbf5e85366 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:16:23.606412 14579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92a389a3e45944bf836899bbf5e85366 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "92a389a3e45944bf836899bbf5e85366" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92a389a3e45944bf836899bbf5e85366" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 44387 } } }
I20260924 16:16:23.606854 14579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92a389a3e45944bf836899bbf5e85366 [sys.catalog]: This master's current role is: LEADER
I20260924 16:16:23.608649 14583 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:16:23.608559 14580 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92a389a3e45944bf836899bbf5e85366 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92a389a3e45944bf836899bbf5e85366. Latest consensus state: current_term: 2 leader_uuid: "92a389a3e45944bf836899bbf5e85366" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92a389a3e45944bf836899bbf5e85366" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 44387 } } }
I20260924 16:16:23.608973 14580 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92a389a3e45944bf836899bbf5e85366 [sys.catalog]: This master's current role is: LEADER
I20260924 16:16:23.616690 14583 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:16:23.618125 14583 catalog_manager.cc:1259] Loaded cluster ID: fe4623367c664b69928c0d8cd63e4002
I20260924 16:16:23.618297 14583 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:16:23.622543 14583 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:16:23.624238 14583 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 92a389a3e45944bf836899bbf5e85366: Loaded TSK: 0
I20260924 16:16:23.658433 14583 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 92a389a3e45944bf836899bbf5e85366: Generated new TSK 1
I20260924 16:16:23.658893 14583 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 16:16:23.909898 14576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:16:23.910229 14576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:16:23.910396 14576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:16:23.917925 14576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:16:23.918092 14576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:16:23.918272 14576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.1
I20260924 16:16:23.928704 14576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:45871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab
--principal=kudu/127.29.234.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.1:34979
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.234.1
--webserver_port=43803
--tserver_master_addrs=127.29.234.62:44387
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.1
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:16:23.930222 14576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:16:23.931634 14576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:16:23.947361 14601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:16:23.947314 14602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:16:23.947383 14604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:16:23.948307 14576 server_base.cc:1034] running on GCE node
I20260924 16:16:23.948954 14576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:16:23.950078 14576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:16:23.951318 14576 hybrid_clock.cc:648] HybridClock initialized: now 1790266583951253 us; error 54 us; skew 500 ppm
I20260924 16:16:23.951647 14576 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:16:23 dist-test-slave-v5j9 krb5kdc[13940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266583, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:16:23.955158 14576 init.cc:377] Logged in from keytab as kudu/127.29.234.1@KRBTEST.COM (short username kudu)
I20260924 16:16:23.956772 14576 webserver.cc:460] Webserver started at http://127.29.234.1:43803/ using document root <none> and password file <none>
I20260924 16:16:23.957584 14576 fs_manager.cc:359] Metadata directory not provided
I20260924 16:16:23.957747 14576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:16:23.963358 14576 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 16:16:23.966295 14611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:16:23.967736 14576 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:16:23.967911 14576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-0/wal
uuid: "4287e1c923de4b69a9c02d00cf413ae7"
format_stamp: "Formatted at 2026-09-24 16:16:01 on dist-test-slave-v5j9"
server_key: "d473dca69abee598abdf3d378d4d362d"
server_key_iv: "9608e050948fd8d14b4b6cfda4253373"
server_key_version: "encryptionkey@0"
I20260924 16:16:23.968909 14576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:16:23.982678 14576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:16:23.986886 14576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:16:23.987154 14576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:16:23.988122 14576 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:16:23.989501 14576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:16:23.989606 14576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:16:23.989737 14576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:16:23.989836 14576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:16:24.026489 14576 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.1:34979
I20260924 16:16:24.026575 14724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.1:34979 every 8 connection(s)
I20260924 16:16:24.028178 14576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
I20260924 16:16:24.031698 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 14576
I20260924 16:16:24.032534 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.2:37037
--local_ip_for_outbound_sockets=127.29.234.2
--tserver_master_addrs=127.29.234.62:44387
--webserver_port=45995
--webserver_interface=127.29.234.2
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab
--principal=kudu/127.29.234.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:45871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 16:16:24 dist-test-slave-v5j9 krb5kdc[13940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266583, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.1@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:16:24.057494 14728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:24.031886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:44387 (local address 127.29.234.1:55781)
0924 16:16:24.032948 (+ 1062us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:24.033004 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 16:16:24.034989 (+ 1985us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:16:24.036674 (+ 1685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:16:24.036685 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:16:24.037484 (+ 799us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:16:24.038133 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:16:24.038154 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:16:24.040642 (+ 2488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:16:24.040654 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:16:24.042043 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:16:24.042057 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:16:24.043732 (+ 1675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:16:24.043737 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:16:24.043927 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:24.044697 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:16:24.044719 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:16:24.047401 (+ 2682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:16:24.052614 (+ 5213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:16:24.052622 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:16:24.052636 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:16:24.053214 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:16:24.053887 (+ 673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:16:24.053892 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:16:24.053894 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:16:24.054117 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:16:24.055285 (+ 1168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:16:24.055291 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:16:24.055626 (+ 335us) client_negotiation.cc:769] Sending connection context
0924 16:16:24.055872 (+ 246us) client_negotiation.cc:240] Negotiation successful
0924 16:16:24.056356 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"thread_start_us":425,"threads_started":1}
I20260924 16:16:24.058202 14727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:24.031525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.1:55781 (local address 127.29.234.62:44387)
0924 16:16:24.032446 (+ 921us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:24.032470 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:24.033166 (+ 696us) server_negotiation.cc:407] Connection header received
0924 16:16:24.035630 (+ 2464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:24.035645 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:24.036041 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:24.036480 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:24.038366 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:24.040431 (+ 2065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:24.042202 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:24.043490 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:24.043725 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:24.047775 (+ 4050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:16:24.047790 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:16:24.047815 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:16:24.047872 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:16:24.052435 (+ 4563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:24.053426 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:24.053439 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:24.053452 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:24.053727 (+ 275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:24.054283 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:24.054292 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:24.054298 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:24.055124 (+ 826us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:16:24.055363 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:24.056206 (+ 843us) server_negotiation.cc:299] Negotiation successful
0924 16:16:24.056768 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":401,"threads_started":1}
I20260924 16:16:24.061710 14725 heartbeater.cc:347] Connected to a master server at 127.29.234.62:44387
I20260924 16:16:24.062079 14725 heartbeater.cc:464] Registering TS with master...
I20260924 16:16:24.062909 14725 heartbeater.cc:511] Master 127.29.234.62:44387 requested a full tablet report, sending...
I20260924 16:16:24.065486 14538 ts_manager.cc:194] Registered new tserver with Master: 4287e1c923de4b69a9c02d00cf413ae7 (127.29.234.1:34979)
I20260924 16:16:24.067245 14538 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.234.1@KRBTEST.COM'} at 127.29.234.1:55781
W20260924 16:16:24.375514 14729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:16:24.375852 14729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:16:24.376030 14729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:16:24.383213 14729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:16:24.383370 14729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:16:24.383550 14729 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.2
I20260924 16:16:24.393458 14729 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:45871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab
--principal=kudu/127.29.234.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.2:37037
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.234.2
--webserver_port=45995
--tserver_master_addrs=127.29.234.62:44387
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.2
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:16:24.394909 14729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:16:24.396354 14729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:16:24.410328 14734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:16:24.410369 14737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:16:24.410822 14735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:16:24.411341 14729 server_base.cc:1034] running on GCE node
I20260924 16:16:24.411957 14729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:16:24.413007 14729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:16:24.414243 14729 hybrid_clock.cc:648] HybridClock initialized: now 1790266584414196 us; error 38 us; skew 500 ppm
I20260924 16:16:24.414573 14729 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:16:24 dist-test-slave-v5j9 krb5kdc[13940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266584, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:16:24.417837 14729 init.cc:377] Logged in from keytab as kudu/127.29.234.2@KRBTEST.COM (short username kudu)
I20260924 16:16:24.419315 14729 webserver.cc:460] Webserver started at http://127.29.234.2:45995/ using document root <none> and password file <none>
I20260924 16:16:24.420028 14729 fs_manager.cc:359] Metadata directory not provided
I20260924 16:16:24.420189 14729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:16:24.425808 14729 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260924 16:16:24.428439 14744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:16:24.429838 14729 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:16:24.430012 14729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-1/wal
uuid: "ad7a4fa907754ea2849edb3d6089728f"
format_stamp: "Formatted at 2026-09-24 16:16:01 on dist-test-slave-v5j9"
server_key: "a01b3e512cad3af12263c7e771a4c6c3"
server_key_iv: "19a2275bc8b410cee47e3aa9552f924b"
server_key_version: "encryptionkey@0"
I20260924 16:16:24.430976 14729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:16:24.443955 14729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:16:24.448139 14729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:16:24.448431 14729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:16:24.449497 14729 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:16:24.450903 14729 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:16:24.451030 14729 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:16:24.451191 14729 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:16:24.451282 14729 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:16:24.484118 14729 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.2:37037
I20260924 16:16:24.484174 14860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.2:37037 every 8 connection(s)
I20260924 16:16:24.485749 14729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
I20260924 16:16:24.491056 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 14729
I20260924 16:16:24.491667 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.3:40389
--local_ip_for_outbound_sockets=127.29.234.3
--tserver_master_addrs=127.29.234.62:44387
--webserver_port=43237
--webserver_interface=127.29.234.3
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab
--principal=kudu/127.29.234.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:45871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 16:16:24 dist-test-slave-v5j9 krb5kdc[13940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266584, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.2@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:16:24.508735 14727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:24.488338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.2:36627 (local address 127.29.234.62:44387)
0924 16:16:24.488516 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:24.488520 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:24.489722 (+ 1202us) server_negotiation.cc:407] Connection header received
0924 16:16:24.491652 (+ 1930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:24.491657 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:24.491752 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:24.491869 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:24.493539 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:24.494231 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:24.495907 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:24.496362 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:24.496503 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:24.500685 (+ 4182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:16:24.500691 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:16:24.500694 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:16:24.500757 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:16:24.504473 (+ 3716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:24.505288 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:24.505297 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:24.505303 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:24.505502 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:24.505987 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:24.505995 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:24.506000 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:24.507029 (+ 1029us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:16:24.507192 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:24.507946 (+ 754us) server_negotiation.cc:299] Negotiation successful
0924 16:16:24.508246 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 16:16:24.509575 14863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:24.488809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:44387 (local address 127.29.234.2:36627)
0924 16:16:24.489560 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:24.489603 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 16:16:24.491367 (+ 1764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:16:24.492211 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:16:24.492219 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:16:24.492730 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:16:24.493313 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:16:24.493331 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:16:24.494398 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:16:24.494404 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:16:24.495740 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:16:24.495752 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:16:24.496597 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:16:24.496604 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:16:24.496842 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:24.497566 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:16:24.497589 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:16:24.500351 (+ 2762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:16:24.504633 (+ 4282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:16:24.504640 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:16:24.504655 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:16:24.505140 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:16:24.505637 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:16:24.505642 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:16:24.505645 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:16:24.505848 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:16:24.507193 (+ 1345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:16:24.507199 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:16:24.507545 (+ 346us) client_negotiation.cc:769] Sending connection context
0924 16:16:24.507783 (+ 238us) client_negotiation.cc:240] Negotiation successful
0924 16:16:24.508193 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":263,"threads_started":1}
I20260924 16:16:24.511189 14861 heartbeater.cc:347] Connected to a master server at 127.29.234.62:44387
I20260924 16:16:24.511534 14861 heartbeater.cc:464] Registering TS with master...
I20260924 16:16:24.512284 14861 heartbeater.cc:511] Master 127.29.234.62:44387 requested a full tablet report, sending...
I20260924 16:16:24.513958 14538 ts_manager.cc:194] Registered new tserver with Master: ad7a4fa907754ea2849edb3d6089728f (127.29.234.2:37037)
I20260924 16:16:24.515340 14538 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.234.2@KRBTEST.COM'} at 127.29.234.2:36627
W20260924 16:16:24.834458 14864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:16:24.834766 14864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:16:24.834865 14864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:16:24.842588 14864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:16:24.842715 14864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:16:24.842830 14864 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.3
I20260924 16:16:24.853216 14864 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:45871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab
--principal=kudu/127.29.234.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.3:40389
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.234.3
--webserver_port=43237
--tserver_master_addrs=127.29.234.62:44387
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.3
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:16:24.854636 14864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:16:24.855993 14864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:16:24.870211 14870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:16:24.870191 14869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:16:24.870252 14872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:16:24.871145 14864 server_base.cc:1034] running on GCE node
I20260924 16:16:24.871788 14864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:16:24.873309 14864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:16:24.874560 14864 hybrid_clock.cc:648] HybridClock initialized: now 1790266584874502 us; error 53 us; skew 500 ppm
I20260924 16:16:24.874869 14864 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:16:24 dist-test-slave-v5j9 krb5kdc[13940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266584, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:16:24.878048 14864 init.cc:377] Logged in from keytab as kudu/127.29.234.3@KRBTEST.COM (short username kudu)
I20260924 16:16:24.879655 14864 webserver.cc:460] Webserver started at http://127.29.234.3:43237/ using document root <none> and password file <none>
I20260924 16:16:24.880406 14864 fs_manager.cc:359] Metadata directory not provided
I20260924 16:16:24.880568 14864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:16:24.886340 14864 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.002s
I20260924 16:16:24.889308 14879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:16:24.890892 14864 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 16:16:24.891091 14864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-2/wal
uuid: "0bef0b8e4f654d14bff64104add0e48b"
format_stamp: "Formatted at 2026-09-24 16:16:02 on dist-test-slave-v5j9"
server_key: "74d76fcce6e7c948021a25b67690f66b"
server_key_iv: "62210afdaef59af2c17ccd22fc60e4a2"
server_key_version: "encryptionkey@0"
I20260924 16:16:24.892196 14864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:16:24.922438 14864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:16:24.927076 14864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:16:24.927388 14864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:16:24.928442 14864 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:16:24.929893 14864 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:16:24.930029 14864 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:16:24.930166 14864 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:16:24.930256 14864 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:16:24.964651 14864 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.3:40389
I20260924 16:16:24.964737 14992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.3:40389 every 8 connection(s)
I20260924 16:16:24.966202 14864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
I20260924 16:16:24.971517 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 14864
Sep 24 16:16:24 dist-test-slave-v5j9 krb5kdc[13940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266584, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.3@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:16:24.988101 14996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:24.972555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37830 (local address 127.29.234.62:44387)
0924 16:16:24.973080 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:24.973088 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:24.973560 (+ 472us) server_negotiation.cc:407] Connection header received
0924 16:16:24.973684 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:24.973709 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:24.973789 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:24.973894 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:24.975794 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:24.976601 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:24.978086 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:24.978561 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:24.978712 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:24.981856 (+ 3144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:16:24.981862 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:16:24.981865 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:16:24.981897 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:16:24.985105 (+ 3208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:24.985950 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:24.985956 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:24.985960 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:24.986111 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:24.986510 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:24.986516 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:24.986525 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:24.986820 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:16:24.987078 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:24.987449 (+ 371us) server_negotiation.cc:299] Negotiation successful
0924 16:16:24.987698 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":308,"threads_started":1}
I20260924 16:16:24.988998 14727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:24.968876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.3:40317 (local address 127.29.234.62:44387)
0924 16:16:24.969074 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:24.969079 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:24.970299 (+ 1220us) server_negotiation.cc:407] Connection header received
0924 16:16:24.972664 (+ 2365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:24.972670 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:24.972757 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:24.972875 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:24.974945 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:24.975925 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:24.977671 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:24.978147 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:24.978286 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:24.982214 (+ 3928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:16:24.982219 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:16:24.982222 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:16:24.982263 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:16:24.985672 (+ 3409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:24.986454 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:24.986459 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:24.986462 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:24.986591 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:24.987050 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:24.987055 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:24.987059 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:24.987351 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:16:24.987501 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:24.988454 (+ 953us) server_negotiation.cc:299] Negotiation successful
0924 16:16:24.988638 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 16:16:24.990545 14995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:24.969324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:44387 (local address 127.29.234.3:40317)
0924 16:16:24.970086 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:24.970172 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0924 16:16:24.972367 (+ 2195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:16:24.973390 (+ 1023us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:16:24.973403 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:16:24.974180 (+ 777us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:16:24.974775 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:16:24.974793 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:16:24.976124 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:16:24.976130 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:16:24.977495 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:16:24.977519 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:16:24.978387 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:16:24.978397 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:16:24.978677 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:24.979416 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:16:24.979451 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:16:24.981971 (+ 2520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:16:24.985847 (+ 3876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:16:24.985855 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:16:24.985869 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:16:24.986322 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:16:24.986708 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:16:24.986715 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:16:24.986723 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:16:24.986915 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:16:24.987565 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:16:24.987571 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:16:24.987995 (+ 424us) client_negotiation.cc:769] Sending connection context
0924 16:16:24.988353 (+ 358us) client_negotiation.cc:240] Negotiation successful
0924 16:16:24.988950 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":268,"threads_started":1}
I20260924 16:16:24.992065 14993 heartbeater.cc:347] Connected to a master server at 127.29.234.62:44387
I20260924 16:16:24.992417 14993 heartbeater.cc:464] Registering TS with master...
I20260924 16:16:24.993230 14993 heartbeater.cc:511] Master 127.29.234.62:44387 requested a full tablet report, sending...
I20260924 16:16:24.994755 14538 ts_manager.cc:194] Registered new tserver with Master: 0bef0b8e4f654d14bff64104add0e48b (127.29.234.3:40389)
I20260924 16:16:24.995705 14538 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.234.3@KRBTEST.COM'} at 127.29.234.3:40317
I20260924 16:16:25.004024 30632 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:16:25.011626 14727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:25.005130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37834 (local address 127.29.234.62:44387)
0924 16:16:25.005408 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:25.005414 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:25.005975 (+ 561us) server_negotiation.cc:407] Connection header received
0924 16:16:25.006214 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:25.006218 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:25.006367 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:25.006502 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:16:25.006956 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:25.007751 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:25.009570 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:25.010154 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:25.010300 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:25.010845 (+ 545us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:16:25.011066 (+ 221us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0924 16:16:25.011423 (+ 357us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37834: token expired
Metrics: {"server-negotiator.queue_time_us":99}
W20260924 16:16:25.011780 14727 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37834: token expired
W20260924 16:16:25.012409 14460 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.29.234.62:44387)
I20260924 16:16:25.026996 14727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:25.012929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37844 (local address 127.29.234.62:44387)
0924 16:16:25.013085 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:25.013088 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:25.013105 (+ 17us) server_negotiation.cc:407] Connection header received
0924 16:16:25.013249 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:25.013252 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:25.013310 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:25.013484 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:25.015083 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:25.015847 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:25.017261 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:25.017747 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:25.017855 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:25.020789 (+ 2934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:16:25.020794 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:16:25.020797 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:16:25.020832 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:16:25.023566 (+ 2734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:25.024371 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:25.024375 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:25.024377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:25.024512 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:25.024962 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:25.024966 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:25.024968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:25.025715 (+ 747us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:16:25.025952 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:25.026398 (+ 446us) server_negotiation.cc:299] Negotiation successful
0924 16:16:25.026630 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 16:16:25.031425 14538 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37844:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:16:25.039994 14538 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:16:25.066397 15003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:25.052455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:34979 (local address 127.0.0.1:40766)
0924 16:16:25.053600 (+ 1145us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:25.053661 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0924 16:16:25.053844 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:16:25.055286 (+ 1442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:16:25.055293 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:16:25.055323 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:16:25.055789 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:16:25.055809 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:16:25.059332 (+ 3523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:16:25.059341 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:16:25.062640 (+ 3299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:16:25.062662 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:16:25.064992 (+ 2330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:16:25.065001 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:16:25.065319 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:25.065424 (+ 105us) client_negotiation.cc:769] Sending connection context
0924 16:16:25.065565 (+ 141us) client_negotiation.cc:240] Negotiation successful
0924 16:16:25.065846 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":855,"thread_start_us":1206,"threads_started":1}
I20260924 16:16:25.068686 15005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:25.052839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40766 (local address 127.29.234.1:34979)
0924 16:16:25.054522 (+ 1683us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:25.054534 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:25.054571 (+ 37us) server_negotiation.cc:407] Connection header received
0924 16:16:25.054713 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:25.054724 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:25.055036 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:25.056108 (+ 1072us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:16:25.056416 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:25.058421 (+ 2005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:25.062916 (+ 4495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:25.064666 (+ 1750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:25.067718 (+ 3052us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:25.068027 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:25.068112 (+ 85us) server_negotiation.cc:299] Negotiation successful
0924 16:16:25.068342 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1454,"thread_start_us":1635,"threads_started":1}
I20260924 16:16:25.070845 14725 heartbeater.cc:503] Master 127.29.234.62:44387 was elected leader, sending a full tablet report...
I20260924 16:16:25.072847 14660 tablet_service.cc:1467] Processing CreateTablet for tablet 9a5b24cdee684608be0f498efcf3baf7 (DEFAULT_TABLE table=test-table [id=46f40506574b4eb4b1439f4e9b4d58f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:16:25.074573 14660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a5b24cdee684608be0f498efcf3baf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:16:25.075888 15003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:25.054197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.2:37037 (local address 127.0.0.1:52204)
0924 16:16:25.066766 (+ 12569us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:25.066790 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 16:16:25.066911 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:16:25.069131 (+ 2220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:16:25.069136 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:16:25.069163 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:16:25.069332 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:16:25.069372 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:16:25.071389 (+ 2017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:16:25.071395 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:16:25.073196 (+ 1801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:16:25.073216 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:16:25.075092 (+ 1876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:16:25.075098 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:16:25.075301 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:25.075336 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 16:16:25.075433 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 16:16:25.075589 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12436,"thread_start_us":4488,"threads_started":1}
I20260924 16:16:25.076017 15007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:25.053593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52204 (local address 127.29.234.2:37037)
0924 16:16:25.068451 (+ 14858us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:25.068462 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:25.068497 (+ 35us) server_negotiation.cc:407] Connection header received
0924 16:16:25.068614 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:25.068622 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:25.068910 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:25.069131 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:16:25.069547 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:25.071241 (+ 1694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:25.073420 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:25.074918 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:25.075064 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:25.075270 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:25.075498 (+ 228us) server_negotiation.cc:299] Negotiation successful
0924 16:16:25.075677 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14650,"thread_start_us":15722,"threads_started":1}
I20260924 16:16:25.077384 15008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:25.052759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.3:40389 (local address 127.0.0.1:39400)
0924 16:16:25.058978 (+ 6219us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:25.059001 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 16:16:25.059142 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:16:25.069337 (+ 10195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:16:25.069367 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:16:25.069394 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:16:25.069736 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:16:25.069746 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:16:25.073112 (+ 3366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:16:25.073118 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:16:25.074654 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:16:25.074673 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:16:25.076569 (+ 1896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:16:25.076576 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:16:25.076746 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:25.076780 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 16:16:25.076867 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 16:16:25.077003 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6062,"thread_start_us":15252,"threads_started":1}
I20260924 16:16:25.077505 15006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:25.053186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39400 (local address 127.29.234.3:40389)
0924 16:16:25.068761 (+ 15575us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:25.068770 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:25.068801 (+ 31us) server_negotiation.cc:407] Connection header received
0924 16:16:25.068887 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:25.068894 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:25.069103 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:25.069939 (+ 836us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:16:25.070229 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:25.072922 (+ 2693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:25.074852 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:25.076334 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:25.076511 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:25.076759 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:25.076948 (+ 189us) server_negotiation.cc:299] Negotiation successful
0924 16:16:25.077122 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15411,"thread_start_us":15221,"threads_started":1}
I20260924 16:16:25.081180 14927 tablet_service.cc:1467] Processing CreateTablet for tablet 9a5b24cdee684608be0f498efcf3baf7 (DEFAULT_TABLE table=test-table [id=46f40506574b4eb4b1439f4e9b4d58f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:16:25.081434 14794 tablet_service.cc:1467] Processing CreateTablet for tablet 9a5b24cdee684608be0f498efcf3baf7 (DEFAULT_TABLE table=test-table [id=46f40506574b4eb4b1439f4e9b4d58f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:16:25.082592 14927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a5b24cdee684608be0f498efcf3baf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:16:25.082749 14794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a5b24cdee684608be0f498efcf3baf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:16:25.091106 15009 tablet_bootstrap.cc:492] T 9a5b24cdee684608be0f498efcf3baf7 P 4287e1c923de4b69a9c02d00cf413ae7: Bootstrap starting.
I20260924 16:16:25.095417 15009 tablet_bootstrap.cc:654] T 9a5b24cdee684608be0f498efcf3baf7 P 4287e1c923de4b69a9c02d00cf413ae7: Neither blocks nor log segments found. Creating new log.
I20260924 16:16:25.097045 15010 tablet_bootstrap.cc:492] T 9a5b24cdee684608be0f498efcf3baf7 P 0bef0b8e4f654d14bff64104add0e48b: Bootstrap starting.
I20260924 16:16:25.098006 15009 log.cc:824] T 9a5b24cdee684608be0f498efcf3baf7 P 4287e1c923de4b69a9c02d00cf413ae7: Log is configured to *not* fsync() on all Append() calls
I20260924 16:16:25.098054 15011 tablet_bootstrap.cc:492] T 9a5b24cdee684608be0f498efcf3baf7 P ad7a4fa907754ea2849edb3d6089728f: Bootstrap starting.
I20260924 16:16:25.100765 15010 tablet_bootstrap.cc:654] T 9a5b24cdee684608be0f498efcf3baf7 P 0bef0b8e4f654d14bff64104add0e48b: Neither blocks nor log segments found. Creating new log.
I20260924 16:16:25.101513 15011 tablet_bootstrap.cc:654] T 9a5b24cdee684608be0f498efcf3baf7 P ad7a4fa907754ea2849edb3d6089728f: Neither blocks nor log segments found. Creating new log.
I20260924 16:16:25.102015 15009 tablet_bootstrap.cc:492] T 9a5b24cdee684608be0f498efcf3baf7 P 4287e1c923de4b69a9c02d00cf413ae7: No bootstrap required, opened a new log
I20260924 16:16:25.102048 15010 log.cc:824] T 9a5b24cdee684608be0f498efcf3baf7 P 0bef0b8e4f654d14bff64104add0e48b: Log is configured to *not* fsync() on all Append() calls
I20260924 16:16:25.102325 15009 ts_tablet_manager.cc:1397] T 9a5b24cdee684608be0f498efcf3baf7 P 4287e1c923de4b69a9c02d00cf413ae7: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260924 16:16:25.102774 15011 log.cc:824] T 9a5b24cdee684608be0f498efcf3baf7 P ad7a4fa907754ea2849edb3d6089728f: Log is configured to *not* fsync() on all Append() calls
I20260924 16:16:25.105177 15010 tablet_bootstrap.cc:492] T 9a5b24cdee684608be0f498efcf3baf7 P 0bef0b8e4f654d14bff64104add0e48b: No bootstrap required, opened a new log
I20260924 16:16:25.105512 15010 ts_tablet_manager.cc:1397] T 9a5b24cdee684608be0f498efcf3baf7 P 0bef0b8e4f654d14bff64104add0e48b: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260924 16:16:25.108049 15011 tablet_bootstrap.cc:492] T 9a5b24cdee684608be0f498efcf3baf7 P ad7a4fa907754ea2849edb3d6089728f: No bootstrap required, opened a new log
I20260924 16:16:25.108387 15011 ts_tablet_manager.cc:1397] T 9a5b24cdee684608be0f498efcf3baf7 P ad7a4fa907754ea2849edb3d6089728f: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20260924 16:16:25.111544 15009 raft_consensus.cc:348] T 9a5b24cdee684608be0f498efcf3baf7 P 4287e1c923de4b69a9c02d00cf413ae7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4287e1c923de4b69a9c02d00cf413ae7" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 34979 } } peers { permanent_uuid: "0bef0b8e4f654d14bff64104add0e48b" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 40389 } } peers { permanent_uuid: "ad7a4fa907754ea2849edb3d6089728f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 37037 } }
I20260924 16:16:25.111946 15009 raft_consensus.cc:374] T 9a5b24cdee684608be0f498efcf3baf7 P 4287e1c923de4b69a9c02d00cf413ae7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:16:25.112052 15009 raft_consensus.cc:729] T 9a5b24cdee684608be0f498efcf3baf7 P 4287e1c923de4b69a9c02d00cf413ae7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4287e1c923de4b69a9c02d00cf413ae7, State: Initialized, Role: FOLLOWER
I20260924 16:16:25.112709 15009 consensus_queue.cc:260] T 9a5b24cdee684608be0f498efcf3baf7 P 4287e1c923de4b69a9c02d00cf413ae7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4287e1c923de4b69a9c02d00cf413ae7" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 34979 } } peers { permanent_uuid: "0bef0b8e4f654d14bff64104add0e48b" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 40389 } } peers { permanent_uuid: "ad7a4fa907754ea2849edb3d6089728f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 37037 } }
I20260924 16:16:25.114513 15009 ts_tablet_manager.cc:1428] T 9a5b24cdee684608be0f498efcf3baf7 P 4287e1c923de4b69a9c02d00cf413ae7: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260924 16:16:25.116222 15010 raft_consensus.cc:348] T 9a5b24cdee684608be0f498efcf3baf7 P 0bef0b8e4f654d14bff64104add0e48b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4287e1c923de4b69a9c02d00cf413ae7" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 34979 } } peers { permanent_uuid: "0bef0b8e4f654d14bff64104add0e48b" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 40389 } } peers { permanent_uuid: "ad7a4fa907754ea2849edb3d6089728f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 37037 } }
I20260924 16:16:25.116636 15010 raft_consensus.cc:374] T 9a5b24cdee684608be0f498efcf3baf7 P 0bef0b8e4f654d14bff64104add0e48b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:16:25.116739 15010 raft_consensus.cc:729] T 9a5b24cdee684608be0f498efcf3baf7 P 0bef0b8e4f654d14bff64104add0e48b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bef0b8e4f654d14bff64104add0e48b, State: Initialized, Role: FOLLOWER
I20260924 16:16:25.117327 15010 consensus_queue.cc:260] T 9a5b24cdee684608be0f498efcf3baf7 P 0bef0b8e4f654d14bff64104add0e48b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4287e1c923de4b69a9c02d00cf413ae7" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 34979 } } peers { permanent_uuid: "0bef0b8e4f654d14bff64104add0e48b" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 40389 } } peers { permanent_uuid: "ad7a4fa907754ea2849edb3d6089728f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 37037 } }
I20260924 16:16:25.118527 14993 heartbeater.cc:503] Master 127.29.234.62:44387 was elected leader, sending a full tablet report...
I20260924 16:16:25.119206 15010 ts_tablet_manager.cc:1428] T 9a5b24cdee684608be0f498efcf3baf7 P 0bef0b8e4f654d14bff64104add0e48b: Time spent starting tablet: real 0.014s user 0.006s sys 0.007s
I20260924 16:16:25.119113 15011 raft_consensus.cc:348] T 9a5b24cdee684608be0f498efcf3baf7 P ad7a4fa907754ea2849edb3d6089728f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4287e1c923de4b69a9c02d00cf413ae7" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 34979 } } peers { permanent_uuid: "0bef0b8e4f654d14bff64104add0e48b" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 40389 } } peers { permanent_uuid: "ad7a4fa907754ea2849edb3d6089728f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 37037 } }
I20260924 16:16:25.119585 15011 raft_consensus.cc:374] T 9a5b24cdee684608be0f498efcf3baf7 P ad7a4fa907754ea2849edb3d6089728f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:16:25.119719 15011 raft_consensus.cc:729] T 9a5b24cdee684608be0f498efcf3baf7 P ad7a4fa907754ea2849edb3d6089728f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad7a4fa907754ea2849edb3d6089728f, State: Initialized, Role: FOLLOWER
I20260924 16:16:25.120352 15011 consensus_queue.cc:260] T 9a5b24cdee684608be0f498efcf3baf7 P ad7a4fa907754ea2849edb3d6089728f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4287e1c923de4b69a9c02d00cf413ae7" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 34979 } } peers { permanent_uuid: "0bef0b8e4f654d14bff64104add0e48b" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 40389 } } peers { permanent_uuid: "ad7a4fa907754ea2849edb3d6089728f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 37037 } }
I20260924 16:16:25.121524 14861 heartbeater.cc:503] Master 127.29.234.62:44387 was elected leader, sending a full tablet report...
I20260924 16:16:25.122176 15011 ts_tablet_manager.cc:1428] T 9a5b24cdee684608be0f498efcf3baf7 P ad7a4fa907754ea2849edb3d6089728f: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
W20260924 16:16:25.217881 14994 tablet.cc:2369] T 9a5b24cdee684608be0f498efcf3baf7 P 0bef0b8e4f654d14bff64104add0e48b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:16:25.218853 15015 raft_consensus.cc:482] T 9a5b24cdee684608be0f498efcf3baf7 P 4287e1c923de4b69a9c02d00cf413ae7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:16:25.219157 15015 raft_consensus.cc:504] T 9a5b24cdee684608be0f498efcf3baf7 P 4287e1c923de4b69a9c02d00cf413ae7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4287e1c923de4b69a9c02d00cf413ae7" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 34979 } } peers { permanent_uuid: "0bef0b8e4f654d14bff64104add0e48b" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 40389 } } peers { permanent_uuid: "ad7a4fa907754ea2849edb3d6089728f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 37037 } }
I20260924 16:16:25.220938 15015 leader_election.cc:288] T 9a5b24cdee684608be0f498efcf3baf7 P 4287e1c923de4b69a9c02d00cf413ae7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0bef0b8e4f654d14bff64104add0e48b (127.29.234.3:40389), ad7a4fa907754ea2849edb3d6089728f (127.29.234.2:37037)
I20260924 16:16:25.223028 15017 raft_consensus.cc:482] T 9a5b24cdee684608be0f498efcf3baf7 P ad7a4fa907754ea2849edb3d6089728f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:16:25.223330 15017 raft_consensus.cc:504] T 9a5b24cdee684608be0f498efcf3baf7 P ad7a4fa907754ea2849edb3d6089728f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4287e1c923de4b69a9c02d00cf413ae7" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 34979 } } peers { permanent_uuid: "0bef0b8e4f654d14bff64104add0e48b" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 40389 } } peers { permanent_uuid: "ad7a4fa907754ea2849edb3d6089728f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 37037 } }
I20260924 16:16:25.228942 15017 leader_election.cc:288] T 9a5b24cdee684608be0f498efcf3baf7 P ad7a4fa907754ea2849edb3d6089728f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4287e1c923de4b69a9c02d00cf413ae7 (127.29.234.1:34979), 0bef0b8e4f654d14bff64104add0e48b (127.29.234.3:40389)
I20260924 16:16:25.230275 15007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:25.221493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.1:41111 (local address 127.29.234.2:37037)
0924 16:16:25.221736 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:25.221744 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:25.222375 (+ 631us) server_negotiation.cc:407] Connection header received
0924 16:16:25.222666 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:25.222670 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:25.222764 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:25.222853 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:16:25.223610 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:25.224498 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:25.227138 (+ 2640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:25.228531 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:25.228665 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:25.228738 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:25.229776 (+ 1038us) server_negotiation.cc:299] Negotiation successful
0924 16:16:25.229900 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 16:16:25.230319 15018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:25.221249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.2:37037 (local address 127.29.234.1:41111)
0924 16:16:25.222227 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:25.222252 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 16:16:25.222495 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:16:25.222898 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:16:25.222903 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:16:25.222930 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:16:25.223428 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:16:25.223447 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:16:25.224652 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:16:25.224659 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:16:25.226934 (+ 2275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:16:25.226955 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:16:25.229432 (+ 2477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:16:25.229440 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:16:25.229628 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:25.229658 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 16:16:25.229753 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 16:16:25.229895 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":792,"spinlock_wait_cycles":15104,"thread_start_us":548,"threads_started":1}
I20260924 16:16:25.230916 15006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:25.221365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.1:46729 (local address 127.29.234.3:40389)
0924 16:16:25.221610 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:25.221618 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:25.222382 (+ 764us) server_negotiation.cc:407] Connection header received
0924 16:16:25.222808 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:25.222817 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:25.222901 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:25.222983 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:16:25.223787 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:25.224826 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:25.227240 (+ 2414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:25.228549 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:25.228666 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:25.228738 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:25.230523 (+ 1785us) server_negotiation.cc:299] Negotiation successful
0924 16:16:25.230667 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260924 16:16:25.231690 15019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:25.221332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.3:40389 (local address 127.29.234.1:46729)
0924 16:16:25.222270 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:25.222285 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 16:16:25.222549 (+ 264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:16:25.223063 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:16:25.223067 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:16:25.223115 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:16:25.223632 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:16:25.223645 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:16:25.224966 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:16:25.224971 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:16:25.227059 (+ 2088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:16:25.227073 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:16:25.228944 (+ 1871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:16:25.228950 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:16:25.229168 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:25.229206 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 16:16:25.231092 (+ 1886us) client_negotiation.cc:240] Negotiation successful
0924 16:16:25.231258 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":810,"mutex_wait_us":5,"spinlock_wait_cycles":134144,"thread_start_us":641,"threads_started":1}
I20260924 16:16:25.231801 14814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a5b24cdee684608be0f498efcf3baf7" candidate_uuid: "4287e1c923de4b69a9c02d00cf413ae7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad7a4fa907754ea2849edb3d6089728f" is_pre_election: true
I20260924 16:16:25.232407 14814 raft_consensus.cc:2454] T 9a5b24cdee684608be0f498efcf3baf7 P ad7a4fa907754ea2849edb3d6089728f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4287e1c923de4b69a9c02d00cf413ae7 in term 0.
I20260924 16:16:25.232771 14948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a5b24cdee684608be0f498efcf3baf7" candidate_uuid: "4287e1c923de4b69a9c02d00cf413ae7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bef0b8e4f654d14bff64104add0e48b" is_pre_election: true
I20260924 16:16:25.233399 14948 raft_consensus.cc:2454] T 9a5b24cdee684608be0f498efcf3baf7 P 0bef0b8e4f654d14bff64104add0e48b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4287e1c923de4b69a9c02d00cf413ae7 in term 0.
I20260924 16:16:25.233986 14615 leader_election.cc:302] T 9a5b24cdee684608be0f498efcf3baf7 P 4287e1c923de4b69a9c02d00cf413ae7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4287e1c923de4b69a9c02d00cf413ae7, ad7a4fa907754ea2849edb3d6089728f; no voters:
I20260924 16:16:25.234771 15015 raft_consensus.cc:2781] T 9a5b24cdee684608be0f498efcf3baf7 P 4287e1c923de4b69a9c02d00cf413ae7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:16:25.234915 15015 raft_consensus.cc:482] T 9a5b24cdee684608be0f498efcf3baf7 P 4287e1c923de4b69a9c02d00cf413ae7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:16:25.235020 15015 raft_consensus.cc:3037] T 9a5b24cdee684608be0f498efcf3baf7 P 4287e1c923de4b69a9c02d00cf413ae7 [term 0 FOLLOWER]: Advancing to term 1
W20260924 16:16:25.237767 14862 tablet.cc:2369] T 9a5b24cdee684608be0f498efcf3baf7 P ad7a4fa907754ea2849edb3d6089728f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:16:25.238144 15015 raft_consensus.cc:504] T 9a5b24cdee684608be0f498efcf3baf7 P 4287e1c923de4b69a9c02d00cf413ae7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4287e1c923de4b69a9c02d00cf413ae7" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 34979 } } peers { permanent_uuid: "0bef0b8e4f654d14bff64104add0e48b" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 40389 } } peers { permanent_uuid: "ad7a4fa907754ea2849edb3d6089728f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 37037 } }
I20260924 16:16:25.239359 15005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:25.226125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.2:41063 (local address 127.29.234.1:34979)
0924 16:16:25.226313 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:25.226319 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:25.230746 (+ 4427us) server_negotiation.cc:407] Connection header received
0924 16:16:25.230959 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:25.230964 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:25.231047 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:25.231130 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:16:25.232350 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:25.233445 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:25.236613 (+ 3168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:25.237990 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:25.238142 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:25.238217 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:25.238976 (+ 759us) server_negotiation.cc:299] Negotiation successful
0924 16:16:25.239109 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 16:16:25.239429 14814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a5b24cdee684608be0f498efcf3baf7" candidate_uuid: "4287e1c923de4b69a9c02d00cf413ae7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad7a4fa907754ea2849edb3d6089728f"
I20260924 16:16:25.239426 14948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a5b24cdee684608be0f498efcf3baf7" candidate_uuid: "4287e1c923de4b69a9c02d00cf413ae7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bef0b8e4f654d14bff64104add0e48b"
I20260924 16:16:25.239651 14814 raft_consensus.cc:3037] T 9a5b24cdee684608be0f498efcf3baf7 P ad7a4fa907754ea2849edb3d6089728f [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:16:25.239686 14948 raft_consensus.cc:3037] T 9a5b24cdee684608be0f498efcf3baf7 P 0bef0b8e4f654d14bff64104add0e48b [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:16:25.240347 15022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:25.226272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.2:47777 (local address 127.29.234.3:40389)
0924 16:16:25.230143 (+ 3871us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:25.230152 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:25.231387 (+ 1235us) server_negotiation.cc:407] Connection header received
0924 16:16:25.231571 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:25.231577 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:25.231665 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:25.231948 (+ 283us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:16:25.233172 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:25.234440 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:25.236527 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:25.237970 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:25.238142 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:25.238217 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:25.239883 (+ 1666us) server_negotiation.cc:299] Negotiation successful
0924 16:16:25.240042 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3689,"thread_start_us":3525,"threads_started":1}
I20260924 16:16:25.240676 15020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:25.225773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.3:40389 (local address 127.29.234.2:47777)
0924 16:16:25.231263 (+ 5490us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:25.231280 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:16:25.231417 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:16:25.232293 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:16:25.232297 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:16:25.232322 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:16:25.232945 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:16:25.232956 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:16:25.234575 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:16:25.234584 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:16:25.236383 (+ 1799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:16:25.236397 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:16:25.238496 (+ 2099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:16:25.238506 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:16:25.238734 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:25.238800 (+ 66us) client_negotiation.cc:769] Sending connection context
0924 16:16:25.238901 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 16:16:25.239048 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5361,"mutex_wait_us":51,"thread_start_us":3652,"threads_started":1}
I20260924 16:16:25.240720 15021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:25.225666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:34979 (local address 127.29.234.2:41063)
0924 16:16:25.230630 (+ 4964us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:25.230653 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 16:16:25.230828 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:16:25.231695 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:16:25.231701 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:16:25.231734 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:16:25.232167 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:16:25.232188 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:16:25.234081 (+ 1893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:16:25.234086 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:16:25.236393 (+ 2307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:16:25.236408 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:16:25.238543 (+ 2135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:16:25.238550 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:16:25.238737 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:25.238837 (+ 100us) client_negotiation.cc:769] Sending connection context
0924 16:16:25.239532 (+ 695us) client_negotiation.cc:240] Negotiation successful
0924 16:16:25.239701 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3944,"mutex_wait_us":75,"thread_start_us":4663,"threads_started":1}
I20260924 16:16:25.241925 14947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a5b24cdee684608be0f498efcf3baf7" candidate_uuid: "ad7a4fa907754ea2849edb3d6089728f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bef0b8e4f654d14bff64104add0e48b" is_pre_election: true
I20260924 16:16:25.243100 14814 raft_consensus.cc:2454] T 9a5b24cdee684608be0f498efcf3baf7 P ad7a4fa907754ea2849edb3d6089728f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4287e1c923de4b69a9c02d00cf413ae7 in term 1.
I20260924 16:16:25.243100 14948 raft_consensus.cc:2454] T 9a5b24cdee684608be0f498efcf3baf7 P 0bef0b8e4f654d14bff64104add0e48b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4287e1c923de4b69a9c02d00cf413ae7 in term 1.
I20260924 16:16:25.243563 14613 leader_election.cc:302] T 9a5b24cdee684608be0f498efcf3baf7 P 4287e1c923de4b69a9c02d00cf413ae7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0bef0b8e4f654d14bff64104add0e48b, 4287e1c923de4b69a9c02d00cf413ae7, ad7a4fa907754ea2849edb3d6089728f; no voters:
I20260924 16:16:25.243908 15015 leader_election.cc:288] T 9a5b24cdee684608be0f498efcf3baf7 P 4287e1c923de4b69a9c02d00cf413ae7 [CANDIDATE]: Term 1 election: Requested vote from peers 0bef0b8e4f654d14bff64104add0e48b (127.29.234.3:40389), ad7a4fa907754ea2849edb3d6089728f (127.29.234.2:37037)
I20260924 16:16:25.244123 15015 raft_consensus.cc:2781] T 9a5b24cdee684608be0f498efcf3baf7 P 4287e1c923de4b69a9c02d00cf413ae7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:16:25.244290 15015 raft_consensus.cc:686] T 9a5b24cdee684608be0f498efcf3baf7 P 4287e1c923de4b69a9c02d00cf413ae7 [term 1 LEADER]: Becoming Leader. State: Replica: 4287e1c923de4b69a9c02d00cf413ae7, State: Running, Role: LEADER
I20260924 16:16:25.244225 14680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a5b24cdee684608be0f498efcf3baf7" candidate_uuid: "ad7a4fa907754ea2849edb3d6089728f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4287e1c923de4b69a9c02d00cf413ae7" is_pre_election: true
I20260924 16:16:25.244880 15015 consensus_queue.cc:237] T 9a5b24cdee684608be0f498efcf3baf7 P 4287e1c923de4b69a9c02d00cf413ae7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4287e1c923de4b69a9c02d00cf413ae7" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 34979 } } peers { permanent_uuid: "0bef0b8e4f654d14bff64104add0e48b" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 40389 } } peers { permanent_uuid: "ad7a4fa907754ea2849edb3d6089728f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 37037 } }
I20260924 16:16:25.245450 14747 leader_election.cc:302] T 9a5b24cdee684608be0f498efcf3baf7 P ad7a4fa907754ea2849edb3d6089728f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ad7a4fa907754ea2849edb3d6089728f; no voters: 0bef0b8e4f654d14bff64104add0e48b, 4287e1c923de4b69a9c02d00cf413ae7
I20260924 16:16:25.245985 15017 raft_consensus.cc:2726] T 9a5b24cdee684608be0f498efcf3baf7 P ad7a4fa907754ea2849edb3d6089728f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 16:16:25.250480 14536 catalog_manager.cc:5495] T 9a5b24cdee684608be0f498efcf3baf7 P 4287e1c923de4b69a9c02d00cf413ae7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4287e1c923de4b69a9c02d00cf413ae7 (127.29.234.1). New cstate: current_term: 1 leader_uuid: "4287e1c923de4b69a9c02d00cf413ae7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4287e1c923de4b69a9c02d00cf413ae7" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 34979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0bef0b8e4f654d14bff64104add0e48b" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 40389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad7a4fa907754ea2849edb3d6089728f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 37037 } health_report { overall_health: UNKNOWN } } }
W20260924 16:16:25.280790 14726 tablet.cc:2369] T 9a5b24cdee684608be0f498efcf3baf7 P 4287e1c923de4b69a9c02d00cf413ae7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:16:25.311177 15005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:25.305285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40770 (local address 127.29.234.1:34979)
0924 16:16:25.305477 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:25.305486 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:25.305529 (+ 43us) server_negotiation.cc:407] Connection header received
0924 16:16:25.305823 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:25.305827 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:25.305918 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:25.306082 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:16:25.306503 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:25.307275 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:25.308697 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:25.309140 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:25.309289 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:25.309860 (+ 571us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:16:25.310501 (+ 641us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:16:25.310678 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:25.310777 (+ 99us) server_negotiation.cc:299] Negotiation successful
0924 16:16:25.310865 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 16:16:25.319792 14948 raft_consensus.cc:1261] T 9a5b24cdee684608be0f498efcf3baf7 P 0bef0b8e4f654d14bff64104add0e48b [term 1 FOLLOWER]: Refusing update from remote peer 4287e1c923de4b69a9c02d00cf413ae7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:16:25.319842 14814 raft_consensus.cc:1261] T 9a5b24cdee684608be0f498efcf3baf7 P ad7a4fa907754ea2849edb3d6089728f [term 1 FOLLOWER]: Refusing update from remote peer 4287e1c923de4b69a9c02d00cf413ae7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:16:25.320854 15023 consensus_queue.cc:1035] T 9a5b24cdee684608be0f498efcf3baf7 P 4287e1c923de4b69a9c02d00cf413ae7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad7a4fa907754ea2849edb3d6089728f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 37037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:16:25.321255 15015 consensus_queue.cc:1035] T 9a5b24cdee684608be0f498efcf3baf7 P 4287e1c923de4b69a9c02d00cf413ae7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bef0b8e4f654d14bff64104add0e48b" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 40389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:16:25.332696 15027 mvcc.cc:205] Tried to move back new op lower bound from 7332931933459988480 to 7332931933171474432. Current Snapshot: MvccSnapshot[applied={T|T < 7332931933459988480}]
I20260924 16:16:25.337471 15026 mvcc.cc:205] Tried to move back new op lower bound from 7332931933459988480 to 7332931933171474432. Current Snapshot: MvccSnapshot[applied={T|T < 7332931933459988480}]
I20260924 16:16:25.350416 14536 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37844:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:16:25.350749 14536 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37844:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:16:25.354264 14536 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 92a389a3e45944bf836899bbf5e85366: Sending DeleteTablet for 3 replicas of tablet 9a5b24cdee684608be0f498efcf3baf7
I20260924 16:16:25.355487 14927 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a5b24cdee684608be0f498efcf3baf7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:16:25 UTC) from {username='kudu', principal='kudu/127.29.234.62@KRBTEST.COM'} at 127.0.0.1:39400
I20260924 16:16:25.355484 14660 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a5b24cdee684608be0f498efcf3baf7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:16:25 UTC) from {username='kudu', principal='kudu/127.29.234.62@KRBTEST.COM'} at 127.0.0.1:40766
I20260924 16:16:25.355773 14794 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a5b24cdee684608be0f498efcf3baf7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:16:25 UTC) from {username='kudu', principal='kudu/127.29.234.62@KRBTEST.COM'} at 127.0.0.1:52204
I20260924 16:16:25.357066 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 14576
I20260924 16:16:25.367879 15037 tablet_replica.cc:330] T 9a5b24cdee684608be0f498efcf3baf7 P ad7a4fa907754ea2849edb3d6089728f: stopping tablet replica
I20260924 16:16:25.369170 15037 raft_consensus.cc:2229] T 9a5b24cdee684608be0f498efcf3baf7 P ad7a4fa907754ea2849edb3d6089728f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:16:25.370427 15035 tablet_replica.cc:330] T 9a5b24cdee684608be0f498efcf3baf7 P 0bef0b8e4f654d14bff64104add0e48b: stopping tablet replica
I20260924 16:16:25.370656 15037 raft_consensus.cc:2258] T 9a5b24cdee684608be0f498efcf3baf7 P ad7a4fa907754ea2849edb3d6089728f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:16:25.371258 15035 raft_consensus.cc:2229] T 9a5b24cdee684608be0f498efcf3baf7 P 0bef0b8e4f654d14bff64104add0e48b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:16:25.371773 15035 raft_consensus.cc:2258] T 9a5b24cdee684608be0f498efcf3baf7 P 0bef0b8e4f654d14bff64104add0e48b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:16:25.373461 15037 ts_tablet_manager.cc:1905] T 9a5b24cdee684608be0f498efcf3baf7 P ad7a4fa907754ea2849edb3d6089728f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:16:25.374363 15035 ts_tablet_manager.cc:1905] T 9a5b24cdee684608be0f498efcf3baf7 P 0bef0b8e4f654d14bff64104add0e48b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:16:25.377453 15036 tablet_replica.cc:330] T 9a5b24cdee684608be0f498efcf3baf7 P 4287e1c923de4b69a9c02d00cf413ae7: stopping tablet replica
I20260924 16:16:25.377839 15005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:25.358008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40776 (local address 127.29.234.1:34979)
0924 16:16:25.358205 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:25.358212 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:25.358416 (+ 204us) server_negotiation.cc:407] Connection header received
0924 16:16:25.358630 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:25.358636 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:25.358749 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:25.358858 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:25.361129 (+ 2271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:25.362339 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:25.364845 (+ 2506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:25.365567 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:25.365723 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:25.369853 (+ 4130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:16:25.369869 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:16:25.369891 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:16:25.369938 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:16:25.374415 (+ 4477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:25.375344 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:25.375354 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:25.375358 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:25.375516 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:25.375961 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:25.375967 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:25.375971 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:25.376521 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:16:25.376832 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:25.377168 (+ 336us) server_negotiation.cc:299] Negotiation successful
0924 16:16:25.377469 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 16:16:25.378755 14720 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:16:25.378006 15036 raft_consensus.cc:2229] T 9a5b24cdee684608be0f498efcf3baf7 P 4287e1c923de4b69a9c02d00cf413ae7 [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:16:25.382510 15037 ts_tablet_manager.cc:1918] T 9a5b24cdee684608be0f498efcf3baf7 P ad7a4fa907754ea2849edb3d6089728f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:16:25.382822 15037 log.cc:1196] T 9a5b24cdee684608be0f498efcf3baf7 P ad7a4fa907754ea2849edb3d6089728f: Deleting WAL directory at /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-1/wal/wals/9a5b24cdee684608be0f498efcf3baf7
I20260924 16:16:25.383427 15037 ts_tablet_manager.cc:1939] T 9a5b24cdee684608be0f498efcf3baf7 P ad7a4fa907754ea2849edb3d6089728f: Deleting consensus metadata
I20260924 16:16:25.383875 15035 ts_tablet_manager.cc:1918] T 9a5b24cdee684608be0f498efcf3baf7 P 0bef0b8e4f654d14bff64104add0e48b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:16:25.384037 15035 log.cc:1196] T 9a5b24cdee684608be0f498efcf3baf7 P 0bef0b8e4f654d14bff64104add0e48b: Deleting WAL directory at /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-2/wal/wals/9a5b24cdee684608be0f498efcf3baf7
I20260924 16:16:25.384578 15035 ts_tablet_manager.cc:1939] T 9a5b24cdee684608be0f498efcf3baf7 P 0bef0b8e4f654d14bff64104add0e48b: Deleting consensus metadata
I20260924 16:16:25.385529 14525 catalog_manager.cc:4841] TS ad7a4fa907754ea2849edb3d6089728f (127.29.234.2:37037): tablet 9a5b24cdee684608be0f498efcf3baf7 (table test-table [id=46f40506574b4eb4b1439f4e9b4d58f9]) successfully deleted
I20260924 16:16:25.386327 14523 catalog_manager.cc:4841] TS 0bef0b8e4f654d14bff64104add0e48b (127.29.234.3:40389): tablet 9a5b24cdee684608be0f498efcf3baf7 (table test-table [id=46f40506574b4eb4b1439f4e9b4d58f9]) successfully deleted
I20260924 16:16:25.479803 15036 raft_consensus.cc:2258] T 9a5b24cdee684608be0f498efcf3baf7 P 4287e1c923de4b69a9c02d00cf413ae7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:16:25.480394 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 14576
I20260924 16:16:25.500540 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 14729
W20260924 16:16:25.500970 14524 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.234.1:34979 (error 108)
I20260924 16:16:25.502785 15008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:25.502364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:34979 (local address 127.0.0.1:40790)
0924 16:16:25.502550 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:25.502644 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.1:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260924 16:16:25.503014 14524 catalog_manager.cc:4584] TS 4287e1c923de4b69a9c02d00cf413ae7 (127.29.234.1:34979): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9a5b24cdee684608be0f498efcf3baf7: Network error: Client connection negotiation failed: client connection to 127.29.234.1:34979: connect: Connection refused (error 111)
I20260924 16:16:25.519398 15007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:25.501513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52206 (local address 127.29.234.2:37037)
0924 16:16:25.501731 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:25.501739 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:25.501762 (+ 23us) server_negotiation.cc:407] Connection header received
0924 16:16:25.501833 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:25.501839 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:25.501947 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:25.502033 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:25.503590 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:25.504374 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:25.505880 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:25.506318 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:25.506415 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:25.513577 (+ 7162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:16:25.513592 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:16:25.513610 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:16:25.513658 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:16:25.516443 (+ 2785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:25.517224 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:25.517233 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:25.517235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:25.517428 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:25.517840 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:25.517845 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:25.517847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:25.518264 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:16:25.518505 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:25.518880 (+ 375us) server_negotiation.cc:299] Negotiation successful
0924 16:16:25.519112 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260924 16:16:25.520098 14855 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:16:25.556535 15008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:25.555973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:34979 (local address 127.0.0.1:40802)
0924 16:16:25.556222 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:25.556343 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.1:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260924 16:16:25.557682 15008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:25.557248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:34979 (local address 127.0.0.1:40818)
0924 16:16:25.557433 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:25.557521 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.1:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260924 16:16:25.618510 15008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:25.617908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:34979 (local address 127.0.0.1:40824)
0924 16:16:25.618136 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:25.618299 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.1:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260924 16:16:25.618947 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 14729
I20260924 16:16:25.620311 15008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:25.619846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:34979 (local address 127.0.0.1:40838)
0924 16:16:25.620021 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:25.620128 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.1:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 16:16:25.638687 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 14864
I20260924 16:16:25.652650 15022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:25.639485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39404 (local address 127.29.234.3:40389)
0924 16:16:25.639688 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:25.639693 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:25.639713 (+ 20us) server_negotiation.cc:407] Connection header received
0924 16:16:25.639876 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:25.639881 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:25.639970 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:25.640058 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:25.641329 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:25.642114 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:25.643405 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:25.643813 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:25.643914 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:25.646585 (+ 2671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:16:25.646603 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:16:25.646632 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:16:25.646668 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:16:25.649446 (+ 2778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:25.650184 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:25.650192 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:25.650195 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:25.650353 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:25.650789 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:25.650792 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:25.650794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:25.651298 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:16:25.651636 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:25.651996 (+ 360us) server_negotiation.cc:299] Negotiation successful
0924 16:16:25.652284 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 16:16:25.653322 14988 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:16:25.701207 15008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:25.700561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:34979 (local address 127.0.0.1:40848)
0924 16:16:25.700867 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:25.700989 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.1:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260924 16:16:25.702447 15008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:25.701987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:34979 (local address 127.0.0.1:40850)
0924 16:16:25.702221 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:25.702312 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.1:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260924 16:16:25.745038 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 14864
I20260924 16:16:25.762646 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 14505
I20260924 16:16:25.763914 14569 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:16:25.883649 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 14505
I20260924 16:16:45.907294 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:44387
--webserver_interface=127.29.234.62
--webserver_port=41217
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab
--principal=kudu/127.29.234.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:45871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.234.62:44387
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:16:46.243395 15043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:16:46.243681 15043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:16:46.243772 15043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:16:46.250993 15043 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 16:16:46.251102 15043 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 16:16:46.251139 15043 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 16:16:46.251181 15043 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:16:46.251225 15043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:16:46.251291 15043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:16:46.251334 15043 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:16:46.251384 15043 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:16:46.260860 15043 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:45871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.29.234.62:44387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab
--principal=kudu/127.29.234.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:44387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.234.62
--webserver_port=41217
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:16:46.262157 15043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:16:46.263543 15043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:16:46.276332 15048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:16:46.276371 15051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:16:46.276417 15049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:16:46.277602 15043 server_base.cc:1034] running on GCE node
I20260924 16:16:46.278373 15043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:16:46.279721 15043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:16:46.280964 15043 hybrid_clock.cc:648] HybridClock initialized: now 1790266606280891 us; error 53 us; skew 500 ppm
I20260924 16:16:46.281296 15043 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:16:46 dist-test-slave-v5j9 krb5kdc[13940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266606, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:16:46.284435 15043 init.cc:377] Logged in from keytab as kudu/127.29.234.62@KRBTEST.COM (short username kudu)
I20260924 16:16:46.285954 15043 webserver.cc:460] Webserver started at http://127.29.234.62:41217/ using document root <none> and password file <none>
I20260924 16:16:46.286695 15043 fs_manager.cc:359] Metadata directory not provided
I20260924 16:16:46.286830 15043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:16:46.292351 15043 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 16:16:46.295233 15058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:16:46.296772 15043 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 16:16:46.296952 15043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/master-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/master-0/wal
uuid: "92a389a3e45944bf836899bbf5e85366"
format_stamp: "Formatted at 2026-09-24 16:16:00 on dist-test-slave-v5j9"
server_key: "cc5bc5cfa98f40f77611023dcbd525df"
server_key_iv: "e93ec137cc70329d4350bc2009d92071"
server_key_version: "encryptionkey@0"
I20260924 16:16:46.297973 15043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:16:46.321162 15043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:16:46.325266 15043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:16:46.325591 15043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:16:46.345299 15043 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.62:44387
I20260924 16:16:46.345320 15110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.62:44387 every 8 connection(s)
I20260924 16:16:46.346957 15043 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
I20260924 16:16:46.351841 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 15043
I20260924 16:16:46.352859 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.1:34979
--local_ip_for_outbound_sockets=127.29.234.1
--tserver_master_addrs=127.29.234.62:44387
--webserver_port=43803
--webserver_interface=127.29.234.1
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab
--principal=kudu/127.29.234.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:45871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:16:46.354954 15112 sys_catalog.cc:263] Verifying existing consensus state
I20260924 16:16:46.360196 15112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92a389a3e45944bf836899bbf5e85366: Bootstrap starting.
I20260924 16:16:46.381701 15112 log.cc:824] T 00000000000000000000000000000000 P 92a389a3e45944bf836899bbf5e85366: Log is configured to *not* fsync() on all Append() calls
I20260924 16:16:46.397894 15112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92a389a3e45944bf836899bbf5e85366: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 16:16:46.398460 15112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92a389a3e45944bf836899bbf5e85366: Bootstrap complete.
I20260924 16:16:46.408134 15112 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92a389a3e45944bf836899bbf5e85366 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92a389a3e45944bf836899bbf5e85366" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 44387 } }
I20260924 16:16:46.409098 15112 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92a389a3e45944bf836899bbf5e85366 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92a389a3e45944bf836899bbf5e85366, State: Initialized, Role: FOLLOWER
I20260924 16:16:46.409794 15112 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92a389a3e45944bf836899bbf5e85366 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92a389a3e45944bf836899bbf5e85366" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 44387 } }
I20260924 16:16:46.410064 15112 raft_consensus.cc:388] T 00000000000000000000000000000000 P 92a389a3e45944bf836899bbf5e85366 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:16:46.410194 15112 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92a389a3e45944bf836899bbf5e85366 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:16:46.410392 15112 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92a389a3e45944bf836899bbf5e85366 [term 2 FOLLOWER]: Advancing to term 3
I20260924 16:16:46.414269 15112 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92a389a3e45944bf836899bbf5e85366 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92a389a3e45944bf836899bbf5e85366" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 44387 } }
I20260924 16:16:46.414767 15112 leader_election.cc:302] T 00000000000000000000000000000000 P 92a389a3e45944bf836899bbf5e85366 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92a389a3e45944bf836899bbf5e85366; no voters:
I20260924 16:16:46.415478 15112 leader_election.cc:288] T 00000000000000000000000000000000 P 92a389a3e45944bf836899bbf5e85366 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 16:16:46.415622 15116 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92a389a3e45944bf836899bbf5e85366 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 16:16:46.416726 15116 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92a389a3e45944bf836899bbf5e85366 [term 3 LEADER]: Becoming Leader. State: Replica: 92a389a3e45944bf836899bbf5e85366, State: Running, Role: LEADER
I20260924 16:16:46.417277 15116 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92a389a3e45944bf836899bbf5e85366 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92a389a3e45944bf836899bbf5e85366" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 44387 } }
I20260924 16:16:46.417763 15112 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92a389a3e45944bf836899bbf5e85366 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:16:46.420032 15118 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92a389a3e45944bf836899bbf5e85366 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92a389a3e45944bf836899bbf5e85366. Latest consensus state: current_term: 3 leader_uuid: "92a389a3e45944bf836899bbf5e85366" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92a389a3e45944bf836899bbf5e85366" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 44387 } } }
I20260924 16:16:46.420477 15118 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92a389a3e45944bf836899bbf5e85366 [sys.catalog]: This master's current role is: LEADER
I20260924 16:16:46.425733 15117 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92a389a3e45944bf836899bbf5e85366 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "92a389a3e45944bf836899bbf5e85366" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92a389a3e45944bf836899bbf5e85366" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 44387 } } }
I20260924 16:16:46.426124 15117 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92a389a3e45944bf836899bbf5e85366 [sys.catalog]: This master's current role is: LEADER
I20260924 16:16:46.435310 15121 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:16:46.448073 15121 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:16:46.450078 15121 catalog_manager.cc:1259] Loaded cluster ID: fe4623367c664b69928c0d8cd63e4002
I20260924 16:16:46.450213 15121 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:16:46.454896 15133 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 16:16:46.455438 15121 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:16:46.456777 15121 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 92a389a3e45944bf836899bbf5e85366: Loaded TSK: 0
I20260924 16:16:46.456939 15121 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 92a389a3e45944bf836899bbf5e85366: Loaded TSK: 1
F20260924 16:16:46.457859 15121 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fe77de189ad google::LogMessage::Fail() at ??:0
@ 0x7fe77de1b88a google::LogMessage::SendToLog() at ??:0
@ 0x7fe77de184de google::LogMessage::Flush() at ??:0
@ 0x7fe77de19a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fe781b7c7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fe781b6b73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fe7938471c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fe793846783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fe7938d22a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7fe7938d20c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fe79391c5e9 std::function<>::operator()() at ??:0
@ 0x7fe79384acdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fe793848c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fe7938d0610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fe7938d0439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fe780a64345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fe780a6791c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fe780a67799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fe780a42eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe783d1c6db start_thread at ??:0
@ 0x7fe77ccb261f clone at ??:0
W20260924 16:16:46.746909 15114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:16:46.747313 15114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:16:46.747494 15114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:16:46.755663 15114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:16:46.755812 15114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:16:46.756001 15114 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.1
I20260924 16:16:46.766022 15114 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:45871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab
--principal=kudu/127.29.234.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.1:34979
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.234.1
--webserver_port=43803
--tserver_master_addrs=127.29.234.62:44387
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.1
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:16:46.767448 15114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:16:46.768862 15114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:16:46.782757 15162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:16:46.782895 15160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:16:46.782857 15159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:16:46.783775 15114 server_base.cc:1034] running on GCE node
I20260924 16:16:46.784447 15114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:16:46.785499 15114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:16:46.786741 15114 hybrid_clock.cc:648] HybridClock initialized: now 1790266606786671 us; error 60 us; skew 500 ppm
I20260924 16:16:46.787083 15114 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:16:46 dist-test-slave-v5j9 krb5kdc[13940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266606, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:16:46.798684 15114 init.cc:377] Logged in from keytab as kudu/127.29.234.1@KRBTEST.COM (short username kudu)
I20260924 16:16:46.800483 15114 webserver.cc:460] Webserver started at http://127.29.234.1:43803/ using document root <none> and password file <none>
I20260924 16:16:46.801298 15114 fs_manager.cc:359] Metadata directory not provided
I20260924 16:16:46.801481 15114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:16:46.807192 15114 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 16:16:46.811841 15169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:16:46.813423 15114 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 16:16:46.813623 15114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-0/wal
uuid: "4287e1c923de4b69a9c02d00cf413ae7"
format_stamp: "Formatted at 2026-09-24 16:16:01 on dist-test-slave-v5j9"
server_key: "d473dca69abee598abdf3d378d4d362d"
server_key_iv: "9608e050948fd8d14b4b6cfda4253373"
server_key_version: "encryptionkey@0"
I20260924 16:16:46.814607 15114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:16:46.831272 15114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:16:46.835403 15114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:16:46.835683 15114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:16:46.836807 15114 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:16:46.838927 15176 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260924 16:16:46.845021 15114 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 16:16:46.845152 15114 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260924 16:16:46.845322 15114 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 16:16:46.848572 15114 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 16:16:46.848651 15114 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20260924 16:16:46.849032 15176 tablet_bootstrap.cc:492] T 9a5b24cdee684608be0f498efcf3baf7 P 4287e1c923de4b69a9c02d00cf413ae7: Bootstrap starting.
I20260924 16:16:46.877665 15176 log.cc:824] T 9a5b24cdee684608be0f498efcf3baf7 P 4287e1c923de4b69a9c02d00cf413ae7: Log is configured to *not* fsync() on all Append() calls
I20260924 16:16:46.891088 15176 tablet_bootstrap.cc:492] T 9a5b24cdee684608be0f498efcf3baf7 P 4287e1c923de4b69a9c02d00cf413ae7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 16:16:46.891434 15114 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.1:34979
I20260924 16:16:46.891460 15286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.1:34979 every 8 connection(s)
I20260924 16:16:46.891839 15176 tablet_bootstrap.cc:492] T 9a5b24cdee684608be0f498efcf3baf7 P 4287e1c923de4b69a9c02d00cf413ae7: Bootstrap complete.
I20260924 16:16:46.892781 15114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
I20260924 16:16:46.893216 15176 ts_tablet_manager.cc:1397] T 9a5b24cdee684608be0f498efcf3baf7 P 4287e1c923de4b69a9c02d00cf413ae7: Time spent bootstrapping tablet: real 0.044s user 0.032s sys 0.008s
I20260924 16:16:46.898516 15289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:46.896387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:44387 (local address 127.29.234.1:56363)
0924 16:16:46.897294 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:46.897603 (+ 309us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":403,"threads_started":1}
W20260924 16:16:46.899101 15287 heartbeater.cc:644] Failed to heartbeat to 127.29.234.62:44387 (0 consecutive failures): Network error: Failed to ping master at 127.29.234.62:44387: Client connection negotiation failed: client connection to 127.29.234.62:44387: connect: Connection refused (error 111)
I20260924 16:16:46.899956 15289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:46.899659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:44387 (local address 127.29.234.1:42721)
0924 16:16:46.899792 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:46.899856 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260924 16:16:46.900890 15289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:46.900561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:44387 (local address 127.29.234.1:47757)
0924 16:16:46.900686 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:46.900762 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 16:16:46.901187 15287 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 16:16:46.902063 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 15114
I20260924 16:16:46.902853 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.2:37037
--local_ip_for_outbound_sockets=127.29.234.2
--tserver_master_addrs=127.29.234.62:44387
--webserver_port=45995
--webserver_interface=127.29.234.2
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab
--principal=kudu/127.29.234.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:45871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:16:46.902827 15176 raft_consensus.cc:348] T 9a5b24cdee684608be0f498efcf3baf7 P 4287e1c923de4b69a9c02d00cf413ae7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4287e1c923de4b69a9c02d00cf413ae7" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 34979 } } peers { permanent_uuid: "0bef0b8e4f654d14bff64104add0e48b" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 40389 } } peers { permanent_uuid: "ad7a4fa907754ea2849edb3d6089728f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 37037 } }
I20260924 16:16:46.904238 15176 raft_consensus.cc:729] T 9a5b24cdee684608be0f498efcf3baf7 P 4287e1c923de4b69a9c02d00cf413ae7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4287e1c923de4b69a9c02d00cf413ae7, State: Initialized, Role: FOLLOWER
I20260924 16:16:46.904883 15176 consensus_queue.cc:260] T 9a5b24cdee684608be0f498efcf3baf7 P 4287e1c923de4b69a9c02d00cf413ae7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4287e1c923de4b69a9c02d00cf413ae7" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 34979 } } peers { permanent_uuid: "0bef0b8e4f654d14bff64104add0e48b" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 40389 } } peers { permanent_uuid: "ad7a4fa907754ea2849edb3d6089728f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 37037 } }
I20260924 16:16:46.906471 15289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:46.906143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:44387 (local address 127.29.234.1:55957)
0924 16:16:46.906305 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:46.906370 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260924 16:16:46.906664 15176 ts_tablet_manager.cc:1428] T 9a5b24cdee684608be0f498efcf3baf7 P 4287e1c923de4b69a9c02d00cf413ae7: Time spent starting tablet: real 0.013s user 0.009s sys 0.008s
I20260924 16:16:46.907507 15289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:46.907203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:44387 (local address 127.29.234.1:37117)
0924 16:16:46.907336 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:46.907401 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260924 16:16:47.240952 15291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:16:47.241281 15291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:16:47.241472 15291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:16:47.248749 15291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:16:47.248896 15291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:16:47.249073 15291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.2
I20260924 16:16:47.258620 15291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:45871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab
--principal=kudu/127.29.234.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.2:37037
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.234.2
--webserver_port=45995
--tserver_master_addrs=127.29.234.62:44387
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.2
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:16:47.259861 15291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:16:47.261217 15291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:16:47.274566 15296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:16:47.274503 15299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:16:47.274533 15297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:16:47.275784 15291 server_base.cc:1034] running on GCE node
I20260924 16:16:47.276324 15291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:16:47.277231 15291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:16:47.278430 15291 hybrid_clock.cc:648] HybridClock initialized: now 1790266607278388 us; error 35 us; skew 500 ppm
I20260924 16:16:47.278697 15291 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:16:47 dist-test-slave-v5j9 krb5kdc[13940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266607, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:16:47.281659 15291 init.cc:377] Logged in from keytab as kudu/127.29.234.2@KRBTEST.COM (short username kudu)
I20260924 16:16:47.282989 15291 webserver.cc:460] Webserver started at http://127.29.234.2:45995/ using document root <none> and password file <none>
I20260924 16:16:47.283686 15291 fs_manager.cc:359] Metadata directory not provided
I20260924 16:16:47.283864 15291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:16:47.289333 15291 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:16:47.292003 15306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:16:47.293287 15291 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 16:16:47.293493 15291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-1/wal
uuid: "ad7a4fa907754ea2849edb3d6089728f"
format_stamp: "Formatted at 2026-09-24 16:16:01 on dist-test-slave-v5j9"
server_key: "a01b3e512cad3af12263c7e771a4c6c3"
server_key_iv: "19a2275bc8b410cee47e3aa9552f924b"
server_key_version: "encryptionkey@0"
I20260924 16:16:47.294502 15291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:16:47.315740 15291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:16:47.320411 15291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:16:47.320701 15291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:16:47.321704 15291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:16:47.323030 15291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:16:47.323165 15291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:16:47.323314 15291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:16:47.323410 15291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:16:47.356902 15291 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.2:37037
I20260924 16:16:47.356957 15420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.2:37037 every 8 connection(s)
I20260924 16:16:47.358477 15291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
I20260924 16:16:47.363345 15423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:47.361528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:44387 (local address 127.29.234.2:38685)
0924 16:16:47.362225 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:47.362503 (+ 278us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":263,"threads_started":1}
W20260924 16:16:47.363969 15421 heartbeater.cc:644] Failed to heartbeat to 127.29.234.62:44387 (0 consecutive failures): Network error: Failed to ping master at 127.29.234.62:44387: Client connection negotiation failed: client connection to 127.29.234.62:44387: connect: Connection refused (error 111)
I20260924 16:16:47.364836 15423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:47.364466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:44387 (local address 127.29.234.2:38847)
0924 16:16:47.364663 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:47.364728 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260924 16:16:47.365674 15423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:47.365332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:44387 (local address 127.29.234.2:43725)
0924 16:16:47.365489 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:47.365570 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260924 16:16:47.365897 15421 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 16:16:47.368716 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 15291
I20260924 16:16:47.369376 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.3:40389
--local_ip_for_outbound_sockets=127.29.234.3
--tserver_master_addrs=127.29.234.62:44387
--webserver_port=43237
--webserver_interface=127.29.234.3
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab
--principal=kudu/127.29.234.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:45871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:16:47.700552 15424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:16:47.700903 15424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:16:47.701077 15424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:16:47.708331 15424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:16:47.708492 15424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:16:47.708675 15424 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.3
I20260924 16:16:47.718341 15424 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:45871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab
--principal=kudu/127.29.234.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.3:40389
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.234.3
--webserver_port=43237
--tserver_master_addrs=127.29.234.62:44387
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.3
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:16:47.719774 15424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:16:47.721150 15424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:16:47.735177 15429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:16:47.735132 15430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:16:47.735234 15432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:16:47.736136 15424 server_base.cc:1034] running on GCE node
I20260924 16:16:47.736759 15424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:16:47.737851 15424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:16:47.739054 15424 hybrid_clock.cc:648] HybridClock initialized: now 1790266607738989 us; error 59 us; skew 500 ppm
I20260924 16:16:47.739383 15424 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:16:47 dist-test-slave-v5j9 krb5kdc[13940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266607, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:16:47.742502 15424 init.cc:377] Logged in from keytab as kudu/127.29.234.3@KRBTEST.COM (short username kudu)
I20260924 16:16:47.743993 15424 webserver.cc:460] Webserver started at http://127.29.234.3:43237/ using document root <none> and password file <none>
I20260924 16:16:47.744685 15424 fs_manager.cc:359] Metadata directory not provided
I20260924 16:16:47.744828 15424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:16:47.750258 15424 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 16:16:47.753005 15439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:16:47.754372 15424 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 16:16:47.754544 15424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-2/wal
uuid: "0bef0b8e4f654d14bff64104add0e48b"
format_stamp: "Formatted at 2026-09-24 16:16:02 on dist-test-slave-v5j9"
server_key: "74d76fcce6e7c948021a25b67690f66b"
server_key_iv: "62210afdaef59af2c17ccd22fc60e4a2"
server_key_version: "encryptionkey@0"
I20260924 16:16:47.755527 15424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:16:47.770362 15424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:16:47.774442 15424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:16:47.774739 15424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:16:47.775746 15424 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:16:47.777149 15424 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:16:47.777253 15424 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:16:47.777407 15424 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:16:47.777509 15424 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:16:47.811928 15424 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.3:40389
I20260924 16:16:47.812021 15552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.3:40389 every 8 connection(s)
I20260924 16:16:47.813496 15424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
I20260924 16:16:47.818348 15555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:47.816451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:44387 (local address 127.29.234.3:42285)
0924 16:16:47.817165 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:47.817504 (+ 339us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":265,"threads_started":1}
W20260924 16:16:47.819075 15553 heartbeater.cc:644] Failed to heartbeat to 127.29.234.62:44387 (0 consecutive failures): Network error: Failed to ping master at 127.29.234.62:44387: Client connection negotiation failed: client connection to 127.29.234.62:44387: connect: Connection refused (error 111)
I20260924 16:16:47.819774 15555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:47.819480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:44387 (local address 127.29.234.3:35631)
0924 16:16:47.819615 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:47.819679 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260924 16:16:47.820541 15555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:47.820262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:44387 (local address 127.29.234.3:60021)
0924 16:16:47.820372 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:47.820438 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260924 16:16:47.820781 15553 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 16:16:47.823107 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 15424
I20260924 16:16:47.909215 15557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:47.908423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:44387 (local address 127.29.234.1:45035)
0924 16:16:47.908884 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:47.908968 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":213,"threads_started":1}
I20260924 16:16:48.348651 15558 raft_consensus.cc:482] T 9a5b24cdee684608be0f498efcf3baf7 P 4287e1c923de4b69a9c02d00cf413ae7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:16:48.348941 15558 raft_consensus.cc:504] T 9a5b24cdee684608be0f498efcf3baf7 P 4287e1c923de4b69a9c02d00cf413ae7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4287e1c923de4b69a9c02d00cf413ae7" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 34979 } } peers { permanent_uuid: "0bef0b8e4f654d14bff64104add0e48b" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 40389 } } peers { permanent_uuid: "ad7a4fa907754ea2849edb3d6089728f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 37037 } }
I20260924 16:16:48.350857 15558 leader_election.cc:288] T 9a5b24cdee684608be0f498efcf3baf7 P 4287e1c923de4b69a9c02d00cf413ae7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0bef0b8e4f654d14bff64104add0e48b (127.29.234.3:40389), ad7a4fa907754ea2849edb3d6089728f (127.29.234.2:37037)
Sep 24 16:16:48 dist-test-slave-v5j9 krb5kdc[13940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266606, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.1@KRBTEST.COM for kudu/127.29.234.2@KRBTEST.COM
Sep 24 16:16:48 dist-test-slave-v5j9 krb5kdc[13940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266606, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.1@KRBTEST.COM for kudu/127.29.234.3@KRBTEST.COM
I20260924 16:16:48.367671 15562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:48.366493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:44387 (local address 127.29.234.2:54909)
0924 16:16:48.367069 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:48.367284 (+ 215us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":242,"threads_started":1}
I20260924 16:16:48.374408 15560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:48.351441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.1:44535 (local address 127.29.234.2:37037)
0924 16:16:48.352184 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:48.352199 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:48.352242 (+ 43us) server_negotiation.cc:407] Connection header received
0924 16:16:48.354237 (+ 1995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:48.354246 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:48.354613 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:48.354952 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:48.356842 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:48.358127 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:48.360286 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:48.361423 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:48.361621 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:48.365889 (+ 4268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:16:48.365897 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:16:48.365912 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:16:48.365975 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:16:48.370336 (+ 4361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:48.371118 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:48.371127 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:48.371135 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:48.371308 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:48.371736 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:48.371742 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:48.371745 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:48.372508 (+ 763us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:16:48.372703 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:48.373650 (+ 947us) server_negotiation.cc:299] Negotiation successful
0924 16:16:48.374013 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":436,"threads_started":1}
I20260924 16:16:48.374418 15561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:48.351617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.1:43341 (local address 127.29.234.3:40389)
0924 16:16:48.352456 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0924 16:16:48.352469 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 16:16:48.352508 (+ 39us) server_negotiation.cc:407] Connection header received
0924 16:16:48.354245 (+ 1737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:16:48.354254 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:16:48.354613 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:16:48.354952 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:16:48.356877 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:48.358710 (+ 1833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:48.360890 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:16:48.361911 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:16:48.362096 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:48.366124 (+ 4028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:16:48.366132 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:16:48.366144 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:16:48.366214 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:16:48.370915 (+ 4701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:48.371569 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:48.371577 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:48.371582 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:48.371733 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:16:48.372098 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:16:48.372102 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:16:48.372105 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:16:48.372745 (+ 640us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:16:48.372943 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0924 16:16:48.373622 (+ 679us) server_negotiation.cc:299] Negotiation successful
0924 16:16:48.373985 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":680,"thread_start_us":924,"threads_started":1}
I20260924 16:16:48.374464 15559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:48.351225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.3:40389 (local address 127.29.234.1:43341)
0924 16:16:48.352045 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:48.352064 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 16:16:48.352565 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:16:48.355077 (+ 2512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:16:48.355088 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:16:48.355889 (+ 801us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:16:48.356651 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:16:48.356686 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:16:48.358917 (+ 2231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:16:48.358922 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:16:48.360430 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:16:48.360444 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:16:48.362125 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:16:48.362132 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:16:48.362315 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:48.363073 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:16:48.363095 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:16:48.365817 (+ 2722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:16:48.371058 (+ 5241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:16:48.371063 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:16:48.371067 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:16:48.371421 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:16:48.371856 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:16:48.371861 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:16:48.371864 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:16:48.371990 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:16:48.372892 (+ 902us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:16:48.372897 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:16:48.373121 (+ 224us) client_negotiation.cc:769] Sending connection context
0924 16:16:48.373427 (+ 306us) client_negotiation.cc:240] Negotiation successful
0924 16:16:48.373996 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":698,"mutex_wait_us":47,"spinlock_wait_cycles":47488,"thread_start_us":335,"threads_started":1}
I20260924 16:16:48.374616 15557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:48.351181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.2:37037 (local address 127.29.234.1:44535)
0924 16:16:48.351966 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:48.352016 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 16:16:48.352562 (+ 546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:16:48.355077 (+ 2515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:16:48.355088 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:16:48.355884 (+ 796us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:16:48.356650 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:16:48.356672 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:16:48.358321 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:16:48.358325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:16:48.360127 (+ 1802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:16:48.360141 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:16:48.361818 (+ 1677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:16:48.361825 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:16:48.362100 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:16:48.362900 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:16:48.362936 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:16:48.365578 (+ 2642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:16:48.370500 (+ 4922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:16:48.370507 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:16:48.370519 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:16:48.370973 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:16:48.371441 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:16:48.371444 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:16:48.371447 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:16:48.371622 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:16:48.372659 (+ 1037us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:16:48.372668 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:16:48.373122 (+ 454us) client_negotiation.cc:769] Sending connection context
0924 16:16:48.373426 (+ 304us) client_negotiation.cc:240] Negotiation successful
0924 16:16:48.374163 (+ 737us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"mutex_wait_us":299,"spinlock_wait_cycles":45568}
W20260924 16:16:48.377799 15171 leader_election.cc:341] T 9a5b24cdee684608be0f498efcf3baf7 P 4287e1c923de4b69a9c02d00cf413ae7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0bef0b8e4f654d14bff64104add0e48b (127.29.234.3:40389): Not found: Tablet not found: 9a5b24cdee684608be0f498efcf3baf7
W20260924 16:16:48.378171 15173 leader_election.cc:341] T 9a5b24cdee684608be0f498efcf3baf7 P 4287e1c923de4b69a9c02d00cf413ae7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad7a4fa907754ea2849edb3d6089728f (127.29.234.2:37037): Not found: Tablet not found: 9a5b24cdee684608be0f498efcf3baf7
I20260924 16:16:48.378366 15173 leader_election.cc:302] T 9a5b24cdee684608be0f498efcf3baf7 P 4287e1c923de4b69a9c02d00cf413ae7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4287e1c923de4b69a9c02d00cf413ae7; no voters: 0bef0b8e4f654d14bff64104add0e48b, ad7a4fa907754ea2849edb3d6089728f
I20260924 16:16:48.378844 15558 raft_consensus.cc:2726] T 9a5b24cdee684608be0f498efcf3baf7 P 4287e1c923de4b69a9c02d00cf413ae7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 16:16:48.822227 15563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:48.821272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:44387 (local address 127.29.234.3:44887)
0924 16:16:48.821861 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:48.822002 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":321,"threads_started":1}
I20260924 16:16:48.911042 15564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:48.910245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:44387 (local address 127.29.234.1:52709)
0924 16:16:48.910741 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:48.910850 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":274,"threads_started":1}
I20260924 16:16:49.369411 15565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:49.368624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:44387 (local address 127.29.234.2:49727)
0924 16:16:49.369114 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:49.369205 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":252,"threads_started":1}
I20260924 16:16:49.823747 15566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:49.823110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:44387 (local address 127.29.234.3:39645)
0924 16:16:49.823524 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:49.823601 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":220,"threads_started":1}
I20260924 16:16:49.912997 15567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:49.912209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:44387 (local address 127.29.234.1:47911)
0924 16:16:49.912718 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:49.912814 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":220,"threads_started":1}
I20260924 16:16:50.112355 15568 raft_consensus.cc:482] T 9a5b24cdee684608be0f498efcf3baf7 P 4287e1c923de4b69a9c02d00cf413ae7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:16:50.112599 15568 raft_consensus.cc:504] T 9a5b24cdee684608be0f498efcf3baf7 P 4287e1c923de4b69a9c02d00cf413ae7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4287e1c923de4b69a9c02d00cf413ae7" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 34979 } } peers { permanent_uuid: "0bef0b8e4f654d14bff64104add0e48b" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 40389 } } peers { permanent_uuid: "ad7a4fa907754ea2849edb3d6089728f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 37037 } }
I20260924 16:16:50.113335 15568 leader_election.cc:288] T 9a5b24cdee684608be0f498efcf3baf7 P 4287e1c923de4b69a9c02d00cf413ae7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0bef0b8e4f654d14bff64104add0e48b (127.29.234.3:40389), ad7a4fa907754ea2849edb3d6089728f (127.29.234.2:37037)
W20260924 16:16:50.114389 15171 leader_election.cc:341] T 9a5b24cdee684608be0f498efcf3baf7 P 4287e1c923de4b69a9c02d00cf413ae7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0bef0b8e4f654d14bff64104add0e48b (127.29.234.3:40389): Not found: Tablet not found: 9a5b24cdee684608be0f498efcf3baf7
W20260924 16:16:50.114605 15173 leader_election.cc:341] T 9a5b24cdee684608be0f498efcf3baf7 P 4287e1c923de4b69a9c02d00cf413ae7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad7a4fa907754ea2849edb3d6089728f (127.29.234.2:37037): Not found: Tablet not found: 9a5b24cdee684608be0f498efcf3baf7
I20260924 16:16:50.114745 15173 leader_election.cc:302] T 9a5b24cdee684608be0f498efcf3baf7 P 4287e1c923de4b69a9c02d00cf413ae7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4287e1c923de4b69a9c02d00cf413ae7; no voters: 0bef0b8e4f654d14bff64104add0e48b, ad7a4fa907754ea2849edb3d6089728f
I20260924 16:16:50.115000 15568 raft_consensus.cc:2726] T 9a5b24cdee684608be0f498efcf3baf7 P 4287e1c923de4b69a9c02d00cf413ae7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 16:16:50.371244 15569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:50.370381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:44387 (local address 127.29.234.2:49407)
0924 16:16:50.370964 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:50.371077 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":314,"threads_started":1}
I20260924 16:16:50.825276 15570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:50.824609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:44387 (local address 127.29.234.3:50925)
0924 16:16:50.825038 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:50.825119 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":229,"threads_started":1}
I20260924 16:16:50.914669 15571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:50.913999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:44387 (local address 127.29.234.1:60979)
0924 16:16:50.914421 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:50.914505 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":223,"threads_started":1}
I20260924 16:16:51.372745 15572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:51.372048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:44387 (local address 127.29.234.2:45905)
0924 16:16:51.372474 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:51.372552 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":201,"threads_started":1}
I20260924 16:16:51.827189 15573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:51.826228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:44387 (local address 127.29.234.3:49927)
0924 16:16:51.826778 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:51.826905 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":228,"threads_started":1}
I20260924 16:16:51.916543 15574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:51.915813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:44387 (local address 127.29.234.1:37945)
0924 16:16:51.916262 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:51.916346 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":226,"threads_started":1}
I20260924 16:16:52.001750 15575 raft_consensus.cc:482] T 9a5b24cdee684608be0f498efcf3baf7 P 4287e1c923de4b69a9c02d00cf413ae7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:16:52.001950 15575 raft_consensus.cc:504] T 9a5b24cdee684608be0f498efcf3baf7 P 4287e1c923de4b69a9c02d00cf413ae7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4287e1c923de4b69a9c02d00cf413ae7" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 34979 } } peers { permanent_uuid: "0bef0b8e4f654d14bff64104add0e48b" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 40389 } } peers { permanent_uuid: "ad7a4fa907754ea2849edb3d6089728f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 37037 } }
I20260924 16:16:52.002566 15575 leader_election.cc:288] T 9a5b24cdee684608be0f498efcf3baf7 P 4287e1c923de4b69a9c02d00cf413ae7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0bef0b8e4f654d14bff64104add0e48b (127.29.234.3:40389), ad7a4fa907754ea2849edb3d6089728f (127.29.234.2:37037)
W20260924 16:16:52.003553 15173 leader_election.cc:341] T 9a5b24cdee684608be0f498efcf3baf7 P 4287e1c923de4b69a9c02d00cf413ae7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad7a4fa907754ea2849edb3d6089728f (127.29.234.2:37037): Not found: Tablet not found: 9a5b24cdee684608be0f498efcf3baf7
W20260924 16:16:52.003794 15171 leader_election.cc:341] T 9a5b24cdee684608be0f498efcf3baf7 P 4287e1c923de4b69a9c02d00cf413ae7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0bef0b8e4f654d14bff64104add0e48b (127.29.234.3:40389): Not found: Tablet not found: 9a5b24cdee684608be0f498efcf3baf7
I20260924 16:16:52.003929 15171 leader_election.cc:302] T 9a5b24cdee684608be0f498efcf3baf7 P 4287e1c923de4b69a9c02d00cf413ae7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4287e1c923de4b69a9c02d00cf413ae7; no voters: 0bef0b8e4f654d14bff64104add0e48b, ad7a4fa907754ea2849edb3d6089728f
I20260924 16:16:52.004206 15575 raft_consensus.cc:2726] T 9a5b24cdee684608be0f498efcf3baf7 P 4287e1c923de4b69a9c02d00cf413ae7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 16:16:52.374580 15576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:52.373887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:44387 (local address 127.29.234.2:35317)
0924 16:16:52.374323 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:52.374405 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":229,"threads_started":1}
I20260924 16:16:52.828892 15577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:52.828152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:44387 (local address 127.29.234.3:51649)
0924 16:16:52.828648 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:52.828730 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":298,"threads_started":1}
I20260924 16:16:52.918349 15578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:52.917574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:44387 (local address 127.29.234.1:38689)
0924 16:16:52.918097 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:52.918193 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":322,"threads_started":1}
I20260924 16:16:53.376343 15579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:53.375604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:44387 (local address 127.29.234.2:38369)
0924 16:16:53.376081 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:53.376170 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":268,"threads_started":1}
I20260924 16:16:53.804754 15580 raft_consensus.cc:482] T 9a5b24cdee684608be0f498efcf3baf7 P 4287e1c923de4b69a9c02d00cf413ae7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:16:53.804987 15580 raft_consensus.cc:504] T 9a5b24cdee684608be0f498efcf3baf7 P 4287e1c923de4b69a9c02d00cf413ae7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4287e1c923de4b69a9c02d00cf413ae7" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 34979 } } peers { permanent_uuid: "0bef0b8e4f654d14bff64104add0e48b" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 40389 } } peers { permanent_uuid: "ad7a4fa907754ea2849edb3d6089728f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 37037 } }
I20260924 16:16:53.805719 15580 leader_election.cc:288] T 9a5b24cdee684608be0f498efcf3baf7 P 4287e1c923de4b69a9c02d00cf413ae7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0bef0b8e4f654d14bff64104add0e48b (127.29.234.3:40389), ad7a4fa907754ea2849edb3d6089728f (127.29.234.2:37037)
W20260924 16:16:53.806862 15173 leader_election.cc:341] T 9a5b24cdee684608be0f498efcf3baf7 P 4287e1c923de4b69a9c02d00cf413ae7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad7a4fa907754ea2849edb3d6089728f (127.29.234.2:37037): Not found: Tablet not found: 9a5b24cdee684608be0f498efcf3baf7
W20260924 16:16:53.807044 15171 leader_election.cc:341] T 9a5b24cdee684608be0f498efcf3baf7 P 4287e1c923de4b69a9c02d00cf413ae7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0bef0b8e4f654d14bff64104add0e48b (127.29.234.3:40389): Not found: Tablet not found: 9a5b24cdee684608be0f498efcf3baf7
I20260924 16:16:53.807142 15171 leader_election.cc:302] T 9a5b24cdee684608be0f498efcf3baf7 P 4287e1c923de4b69a9c02d00cf413ae7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4287e1c923de4b69a9c02d00cf413ae7; no voters: 0bef0b8e4f654d14bff64104add0e48b, ad7a4fa907754ea2849edb3d6089728f
I20260924 16:16:53.807374 15580 raft_consensus.cc:2726] T 9a5b24cdee684608be0f498efcf3baf7 P 4287e1c923de4b69a9c02d00cf413ae7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 16:16:53.830438 15581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:53.829779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:44387 (local address 127.29.234.3:44713)
0924 16:16:53.830193 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:53.830270 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":221,"threads_started":1}
I20260924 16:16:53.919935 15582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:53.919154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:44387 (local address 127.29.234.1:56583)
0924 16:16:53.919615 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:53.919708 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":254,"threads_started":1}
I20260924 16:16:54.378229 15583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:54.377324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:44387 (local address 127.29.234.2:58955)
0924 16:16:54.377941 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:54.378027 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":249,"threads_started":1}
I20260924 16:16:54.832149 15584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:54.831340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:44387 (local address 127.29.234.3:60951)
0924 16:16:54.831856 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:54.831943 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":239,"threads_started":1}
I20260924 16:16:54.921672 15585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:54.920864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:44387 (local address 127.29.234.1:41909)
0924 16:16:54.921410 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:54.921505 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":300,"threads_started":1}
I20260924 16:16:55.379853 15586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:55.379115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:44387 (local address 127.29.234.2:45599)
0924 16:16:55.379544 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:55.379667 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":219,"threads_started":1}
I20260924 16:16:55.833897 15587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:55.833131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:44387 (local address 127.29.234.3:35107)
0924 16:16:55.833608 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:55.833693 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":280,"threads_started":1}
I20260924 16:16:55.923297 15588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:55.922579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:44387 (local address 127.29.234.1:35045)
0924 16:16:55.922999 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:55.923083 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":225,"threads_started":1}
I20260924 16:16:56.381672 15589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:56.380826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:44387 (local address 127.29.234.2:49331)
0924 16:16:56.381425 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:56.381512 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":274,"threads_started":1}
I20260924 16:16:56.835570 15590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:56.834858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:44387 (local address 127.29.234.3:55807)
0924 16:16:56.835333 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:56.835418 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":232,"threads_started":1}
I20260924 16:16:56.925189 15591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:56.924384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:44387 (local address 127.29.234.1:37107)
0924 16:16:56.924931 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:56.925017 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":305,"threads_started":1}
I20260924 16:16:57.383460 15592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:57.382728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:44387 (local address 127.29.234.2:42171)
0924 16:16:57.383206 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:57.383287 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":267,"threads_started":1}
I20260924 16:16:57.837702 15593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:57.836717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:44387 (local address 127.29.234.3:40117)
0924 16:16:57.837293 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:57.837433 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":323,"threads_started":1}
I20260924 16:16:57.926913 15594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:57.926148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:44387 (local address 127.29.234.1:60309)
0924 16:16:57.926622 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:57.926730 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":246,"threads_started":1}
I20260924 16:16:58.385274 15595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:58.384384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:44387 (local address 127.29.234.2:34963)
0924 16:16:58.384954 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:58.385063 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":284,"threads_started":1}
I20260924 16:16:58.839398 15596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:58.838663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:44387 (local address 127.29.234.3:37577)
0924 16:16:58.839128 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:58.839214 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":239,"threads_started":1}
I20260924 16:16:58.928639 15597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:58.927933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:44387 (local address 127.29.234.1:60467)
0924 16:16:58.928391 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:58.928474 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":228,"threads_started":1}
I20260924 16:16:59.387225 15598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:59.386388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:44387 (local address 127.29.234.2:57085)
0924 16:16:59.386977 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:59.387055 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":287,"threads_started":1}
I20260924 16:16:59.842046 15599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:59.840492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:44387 (local address 127.29.234.3:34867)
0924 16:16:59.841675 (+ 1183us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:59.841792 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1042,"thread_start_us":351,"threads_started":1}
I20260924 16:16:59.930517 15600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:16:59.929647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:44387 (local address 127.29.234.1:58233)
0924 16:16:59.930213 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0924 16:16:59.930307 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":291,"threads_started":1}
I20260924 16:16:59.937081 15601 raft_consensus.cc:482] T 9a5b24cdee684608be0f498efcf3baf7 P 4287e1c923de4b69a9c02d00cf413ae7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:16:59.937381 15601 raft_consensus.cc:504] T 9a5b24cdee684608be0f498efcf3baf7 P 4287e1c923de4b69a9c02d00cf413ae7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4287e1c923de4b69a9c02d00cf413ae7" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 34979 } } peers { permanent_uuid: "0bef0b8e4f654d14bff64104add0e48b" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 40389 } } peers { permanent_uuid: "ad7a4fa907754ea2849edb3d6089728f" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 37037 } }
I20260924 16:16:59.938071 15601 leader_election.cc:288] T 9a5b24cdee684608be0f498efcf3baf7 P 4287e1c923de4b69a9c02d00cf413ae7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0bef0b8e4f654d14bff64104add0e48b (127.29.234.3:40389), ad7a4fa907754ea2849edb3d6089728f (127.29.234.2:37037)
W20260924 16:16:59.939060 15173 leader_election.cc:341] T 9a5b24cdee684608be0f498efcf3baf7 P 4287e1c923de4b69a9c02d00cf413ae7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad7a4fa907754ea2849edb3d6089728f (127.29.234.2:37037): Not found: Tablet not found: 9a5b24cdee684608be0f498efcf3baf7
W20260924 16:16:59.939265 15171 leader_election.cc:341] T 9a5b24cdee684608be0f498efcf3baf7 P 4287e1c923de4b69a9c02d00cf413ae7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0bef0b8e4f654d14bff64104add0e48b (127.29.234.3:40389): Not found: Tablet not found: 9a5b24cdee684608be0f498efcf3baf7
I20260924 16:16:59.939404 15171 leader_election.cc:302] T 9a5b24cdee684608be0f498efcf3baf7 P 4287e1c923de4b69a9c02d00cf413ae7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4287e1c923de4b69a9c02d00cf413ae7; no voters: 0bef0b8e4f654d14bff64104add0e48b, ad7a4fa907754ea2849edb3d6089728f
I20260924 16:16:59.939666 15601 raft_consensus.cc:2726] T 9a5b24cdee684608be0f498efcf3baf7 P 4287e1c923de4b69a9c02d00cf413ae7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 16:17:00.388806 15602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:00.388092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:44387 (local address 127.29.234.2:40035)
0924 16:17:00.388529 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0924 16:17:00.388630 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":224,"threads_started":1}
I20260924 16:17:00.843788 15603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:00.843057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:44387 (local address 127.29.234.3:60953)
0924 16:17:00.843524 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0924 16:17:00.843619 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":245,"threads_started":1}
I20260924 16:17:00.932251 15604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:00.931487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:44387 (local address 127.29.234.1:57895)
0924 16:17:00.931926 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0924 16:17:00.932007 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":219,"threads_started":1}
I20260924 16:17:01.390734 15605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:01.389917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:44387 (local address 127.29.234.2:47429)
0924 16:17:01.390408 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0924 16:17:01.390511 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":253,"threads_started":1}
I20260924 16:17:01.854228 15606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:01.844765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:44387 (local address 127.29.234.3:34439)
0924 16:17:01.853798 (+ 9033us) negotiation.cc:107] Waiting for socket to connect
0924 16:17:01.853924 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":8870,"thread_start_us":8696,"threads_started":1}
I20260924 16:17:01.934211 15607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:01.933529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:44387 (local address 127.29.234.1:42415)
0924 16:17:01.933949 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0924 16:17:01.934044 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":212,"threads_started":1}
I20260924 16:17:02.393021 15608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:02.392046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:44387 (local address 127.29.234.2:50535)
0924 16:17:02.392652 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0924 16:17:02.392804 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":269,"threads_started":1}
I20260924 16:17:02.829234 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 15114
I20260924 16:17:02.845063 15609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:02.830471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41402 (local address 127.29.234.1:34979)
0924 16:17:02.831180 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:02.831191 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:02.831227 (+ 36us) server_negotiation.cc:407] Connection header received
0924 16:17:02.831335 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:02.831347 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:02.831695 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:02.831885 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:02.833320 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:02.834389 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:02.836237 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:02.836955 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:02.837056 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:02.839686 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:02.839699 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:02.839716 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:17:02.839756 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:02.842564 (+ 2808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:17:02.843239 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:17:02.843246 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:17:02.843248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:17:02.843397 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:17:02.843742 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:17:02.843745 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:17:02.843747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:17:02.844100 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:02.844340 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:02.844601 (+ 261us) server_negotiation.cc:299] Negotiation successful
0924 16:17:02.844803 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":322,"threads_started":1}
I20260924 16:17:02.846587 15282 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:17:02.856456 15611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:02.855611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:44387 (local address 127.29.234.3:53699)
0924 16:17:02.856154 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0924 16:17:02.856292 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.62:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":317,"threads_started":1}
I20260924 16:17:02.945524 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 15114
I20260924 16:17:02.964176 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 15291
I20260924 16:17:02.977272 15613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:02.964937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41344 (local address 127.29.234.2:37037)
0924 16:17:02.965514 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:02.965519 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:02.965535 (+ 16us) server_negotiation.cc:407] Connection header received
0924 16:17:02.965593 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:02.965596 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:02.965656 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:02.965744 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:02.967109 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:02.967902 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:02.969057 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:02.969532 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:02.969626 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:02.972306 (+ 2680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:02.972311 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:02.972315 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:17:02.972376 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:02.975043 (+ 2667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:17:02.975630 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:17:02.975634 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:17:02.975636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:17:02.975768 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:17:02.976115 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:17:02.976118 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:17:02.976120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:17:02.976364 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:02.976567 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:02.976811 (+ 244us) server_negotiation.cc:299] Negotiation successful
0924 16:17:02.976980 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":330,"threads_started":1}
I20260924 16:17:02.978185 15416 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:17:03.063208 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 15291
I20260924 16:17:03.081663 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 15424
I20260924 16:17:03.095485 15615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:03.082521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35462 (local address 127.29.234.3:40389)
0924 16:17:03.083133 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:03.083138 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:03.083153 (+ 15us) server_negotiation.cc:407] Connection header received
0924 16:17:03.083208 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:03.083211 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:03.083274 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:03.083393 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:03.084758 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:03.085607 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:03.086878 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:03.087296 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:03.087427 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:03.090103 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:03.090108 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:03.090111 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:17:03.090149 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:03.092718 (+ 2569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:17:03.093435 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:17:03.093442 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:17:03.093446 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:17:03.093640 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:17:03.094013 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:17:03.094018 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:17:03.094022 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:17:03.094295 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:03.094570 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:03.094877 (+ 307us) server_negotiation.cc:299] Negotiation successful
0924 16:17:03.095109 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":337,"threads_started":1}
I20260924 16:17:03.096331 15547 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:17:03.181037 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 15424
2026-09-24T16:17:03Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64619 ms)
[----------] 30 tests from SecurityITest (299914 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:17:03 dist-test-slave-v5j9 krb5kdc[15619](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:17:03 dist-test-slave-v5j9 krb5kdc[15619](info): set up 2 sockets
Sep 24 16:17:03 dist-test-slave-v5j9 krb5kdc[15619](info): commencing operation
krb5kdc: starting...
W20260924 16:17:04.321046 30632 mini_kdc.cc:121] Time spent starting KDC: real 1.098s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Sep 24 16:17:04 dist-test-slave-v5j9 krb5kdc[15619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:17:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:17:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.234.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab.
Entry for principal kudu/127.29.234.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab.
I20260924 16:17:04.584697 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:46359
--webserver_interface=127.29.234.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab
--principal=kudu/127.29.234.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:46873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.234.62:46359
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:17:04.923640 15632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:17:04.923936 15632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:17:04.924201 15632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:17:04.931473 15632 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:17:04.931579 15632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:17:04.931689 15632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:17:04.931805 15632 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:17:04.931882 15632 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:17:04.942559 15632 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:46873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.234.62:46359
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab
--principal=kudu/127.29.234.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:46359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.234.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:17:04.943971 15632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:17:04.945493 15632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:17:04.959331 15637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:04.959265 15640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:04.959355 15638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:17:04.959869 15632 server_base.cc:1034] running on GCE node
I20260924 16:17:04.960799 15632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:17:04.962393 15632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:17:04.963671 15632 hybrid_clock.cc:648] HybridClock initialized: now 1790266624963619 us; error 53 us; skew 500 ppm
I20260924 16:17:04.964063 15632 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:17:04 dist-test-slave-v5j9 krb5kdc[15619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266624, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:17:04.967375 15632 init.cc:377] Logged in from keytab as kudu/127.29.234.62@KRBTEST.COM (short username kudu)
I20260924 16:17:04.968981 15632 webserver.cc:460] Webserver started at http://127.29.234.62:33451/ using document root <none> and password file <none>
I20260924 16:17:04.969815 15632 fs_manager.cc:359] Metadata directory not provided
I20260924 16:17:04.969964 15632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:17:04.970300 15632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:17:04.973585 15632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/minicluster-data/master-0/data/instance:
uuid: "2b3ed99e62f840fdb722c5df44e02389"
format_stamp: "Formatted at 2026-09-24 16:17:04 on dist-test-slave-v5j9"
server_key: "2d1aea4ca310846042bac7eb5d0846f7"
server_key_iv: "02161bffac61aad0f6952b77c7b0be03"
server_key_version: "encryptionkey@0"
I20260924 16:17:04.974403 15632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/minicluster-data/master-0/wal/instance:
uuid: "2b3ed99e62f840fdb722c5df44e02389"
format_stamp: "Formatted at 2026-09-24 16:17:04 on dist-test-slave-v5j9"
server_key: "2d1aea4ca310846042bac7eb5d0846f7"
server_key_iv: "02161bffac61aad0f6952b77c7b0be03"
server_key_version: "encryptionkey@0"
I20260924 16:17:04.980486 15632 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260924 16:17:04.984193 15647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:17:04.985935 15632 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:17:04.986181 15632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/minicluster-data/master-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/minicluster-data/master-0/wal
uuid: "2b3ed99e62f840fdb722c5df44e02389"
format_stamp: "Formatted at 2026-09-24 16:17:04 on dist-test-slave-v5j9"
server_key: "2d1aea4ca310846042bac7eb5d0846f7"
server_key_iv: "02161bffac61aad0f6952b77c7b0be03"
server_key_version: "encryptionkey@0"
I20260924 16:17:04.986408 15632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:17:05.008395 15632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:17:05.012687 15632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:17:05.012984 15632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:17:05.032294 15632 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.62:46359
I20260924 16:17:05.032298 15699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.62:46359 every 8 connection(s)
I20260924 16:17:05.033804 15632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
I20260924 16:17:05.037781 15700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:17:05.040839 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 15632
I20260924 16:17:05.041085 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/minicluster-data/master-0/wal/instance
I20260924 16:17:05.041973 30632 external_mini_cluster.cc:1428] Setting key 0730c066893aae4a6890edc177226cdd
I20260924 16:17:05.051270 15700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b3ed99e62f840fdb722c5df44e02389: Bootstrap starting.
I20260924 16:17:05.055401 15700 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b3ed99e62f840fdb722c5df44e02389: Neither blocks nor log segments found. Creating new log.
I20260924 16:17:05.056866 15700 log.cc:824] T 00000000000000000000000000000000 P 2b3ed99e62f840fdb722c5df44e02389: Log is configured to *not* fsync() on all Append() calls
Sep 24 16:17:05 dist-test-slave-v5j9 krb5kdc[15619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:17:05.059968 15700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b3ed99e62f840fdb722c5df44e02389: No bootstrap required, opened a new log
I20260924 16:17:05.066998 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:05.044315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48508 (local address 127.29.234.62:46359)
0924 16:17:05.045136 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:05.045150 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:05.045193 (+ 43us) server_negotiation.cc:407] Connection header received
0924 16:17:05.046984 (+ 1791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:05.047008 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:05.047526 (+ 518us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:05.047931 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:05.049751 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:05.051125 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:05.053053 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:05.053815 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:05.053942 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:05.059356 (+ 5414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:05.059365 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:05.059381 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:17:05.059421 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:05.062721 (+ 3300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:17:05.063481 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:17:05.063488 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:17:05.063495 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:17:05.063668 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:17:05.064063 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:17:05.064068 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:17:05.064073 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:17:05.064639 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:05.064968 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:05.065467 (+ 499us) server_negotiation.cc:299] Negotiation successful
0924 16:17:05.065929 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":389,"threads_started":1}
I20260924 16:17:05.068094 15700 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b3ed99e62f840fdb722c5df44e02389 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b3ed99e62f840fdb722c5df44e02389" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 46359 } }
I20260924 16:17:05.068485 15700 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b3ed99e62f840fdb722c5df44e02389 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:17:05.068617 15700 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b3ed99e62f840fdb722c5df44e02389 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b3ed99e62f840fdb722c5df44e02389, State: Initialized, Role: FOLLOWER
I20260924 16:17:05.069176 15700 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b3ed99e62f840fdb722c5df44e02389 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b3ed99e62f840fdb722c5df44e02389" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 46359 } }
I20260924 16:17:05.069458 15700 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b3ed99e62f840fdb722c5df44e02389 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:17:05.069614 15700 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b3ed99e62f840fdb722c5df44e02389 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:17:05.069810 15700 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b3ed99e62f840fdb722c5df44e02389 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:17:05.072031 15700 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b3ed99e62f840fdb722c5df44e02389 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b3ed99e62f840fdb722c5df44e02389" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 46359 } }
I20260924 16:17:05.072530 15700 leader_election.cc:302] T 00000000000000000000000000000000 P 2b3ed99e62f840fdb722c5df44e02389 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b3ed99e62f840fdb722c5df44e02389; no voters:
I20260924 16:17:05.073096 15700 leader_election.cc:288] T 00000000000000000000000000000000 P 2b3ed99e62f840fdb722c5df44e02389 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:17:05.073266 15705 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b3ed99e62f840fdb722c5df44e02389 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:17:05.074548 15705 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b3ed99e62f840fdb722c5df44e02389 [term 1 LEADER]: Becoming Leader. State: Replica: 2b3ed99e62f840fdb722c5df44e02389, State: Running, Role: LEADER
I20260924 16:17:05.075064 15705 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b3ed99e62f840fdb722c5df44e02389 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b3ed99e62f840fdb722c5df44e02389" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 46359 } }
I20260924 16:17:05.075258 15700 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b3ed99e62f840fdb722c5df44e02389 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:17:05.078110 15707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b3ed99e62f840fdb722c5df44e02389 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b3ed99e62f840fdb722c5df44e02389. Latest consensus state: current_term: 1 leader_uuid: "2b3ed99e62f840fdb722c5df44e02389" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b3ed99e62f840fdb722c5df44e02389" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 46359 } } }
I20260924 16:17:05.078570 15707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b3ed99e62f840fdb722c5df44e02389 [sys.catalog]: This master's current role is: LEADER
I20260924 16:17:05.079017 15706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b3ed99e62f840fdb722c5df44e02389 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b3ed99e62f840fdb722c5df44e02389" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b3ed99e62f840fdb722c5df44e02389" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 46359 } } }
I20260924 16:17:05.079380 15706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b3ed99e62f840fdb722c5df44e02389 [sys.catalog]: This master's current role is: LEADER
I20260924 16:17:05.081202 15711 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:17:05.086746 15711 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:17:05.096254 15711 catalog_manager.cc:1347] Generated new cluster ID: 163f3c63730546eca28a81b8ace9d1d3
I20260924 16:17:05.096513 15711 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:17:05.120245 15711 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:17:05.121788 15711 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:17:05.132048 15711 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b3ed99e62f840fdb722c5df44e02389: Generated new TSK 0
I20260924 16:17:05.132957 15711 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.234.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab.
Entry for principal kudu/127.29.234.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab.
I20260924 16:17:05.198129 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.1:0
--local_ip_for_outbound_sockets=127.29.234.1
--webserver_interface=127.29.234.1
--webserver_port=0
--tserver_master_addrs=127.29.234.62:46359
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab
--principal=kudu/127.29.234.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:46873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:17:05.541934 15726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:17:05.542215 15726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:17:05.542310 15726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:17:05.549296 15726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:17:05.549476 15726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:17:05.549609 15726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.1
I20260924 16:17:05.559270 15726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:46873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab
--principal=kudu/127.29.234.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.234.1
--webserver_port=0
--tserver_master_addrs=127.29.234.62:46359
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.1
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:17:05.560524 15726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:17:05.562009 15726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:17:05.575515 15734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:05.575536 15732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:05.575639 15731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:17:05.581382 15726 server_base.cc:1034] running on GCE node
I20260924 16:17:05.582217 15726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:17:05.583248 15726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:17:05.584463 15726 hybrid_clock.cc:648] HybridClock initialized: now 1790266625584410 us; error 42 us; skew 500 ppm
I20260924 16:17:05.584821 15726 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:17:05 dist-test-slave-v5j9 krb5kdc[15619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266625, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:17:05.588091 15726 init.cc:377] Logged in from keytab as kudu/127.29.234.1@KRBTEST.COM (short username kudu)
I20260924 16:17:05.589766 15726 webserver.cc:460] Webserver started at http://127.29.234.1:42081/ using document root <none> and password file <none>
I20260924 16:17:05.590554 15726 fs_manager.cc:359] Metadata directory not provided
I20260924 16:17:05.590689 15726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:17:05.591027 15726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:17:05.594105 15726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/minicluster-data/ts-0/data/instance:
uuid: "902fcd0c4a5f4498b7d31bc1b28c62da"
format_stamp: "Formatted at 2026-09-24 16:17:05 on dist-test-slave-v5j9"
server_key: "71b2a00a30619928a7a1157b23a89f97"
server_key_iv: "7e9b528c43082556d203f30ae57f74a8"
server_key_version: "encryptionkey@0"
I20260924 16:17:05.594887 15726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/minicluster-data/ts-0/wal/instance:
uuid: "902fcd0c4a5f4498b7d31bc1b28c62da"
format_stamp: "Formatted at 2026-09-24 16:17:05 on dist-test-slave-v5j9"
server_key: "71b2a00a30619928a7a1157b23a89f97"
server_key_iv: "7e9b528c43082556d203f30ae57f74a8"
server_key_version: "encryptionkey@0"
I20260924 16:17:05.600238 15726 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 16:17:05.604161 15741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:17:05.605831 15726 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 16:17:05.606019 15726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/minicluster-data/ts-0/wal
uuid: "902fcd0c4a5f4498b7d31bc1b28c62da"
format_stamp: "Formatted at 2026-09-24 16:17:05 on dist-test-slave-v5j9"
server_key: "71b2a00a30619928a7a1157b23a89f97"
server_key_iv: "7e9b528c43082556d203f30ae57f74a8"
server_key_version: "encryptionkey@0"
I20260924 16:17:05.606266 15726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:17:05.623694 15726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:17:05.627842 15726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:17:05.628135 15726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:17:05.629144 15726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:17:05.630579 15726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:17:05.630656 15726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:17:05.630831 15726 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:17:05.630900 15726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:17:05.664801 15726 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.1:39455
I20260924 16:17:05.664881 15854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.1:39455 every 8 connection(s)
I20260924 16:17:05.666229 15726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
I20260924 16:17:05.673657 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 15726
I20260924 16:17:05.673921 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/minicluster-data/ts-0/wal/instance
I20260924 16:17:05.674610 30632 external_mini_cluster.cc:1428] Setting key 5b988a201a4bb3028d8b3f510982b5bd
Sep 24 16:17:05 dist-test-slave-v5j9 krb5kdc[15619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266625, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.1@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:17:05.687824 15857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:05.668815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.1:56689 (local address 127.29.234.62:46359)
0924 16:17:05.669317 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:05.669321 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:05.670144 (+ 823us) server_negotiation.cc:407] Connection header received
0924 16:17:05.671993 (+ 1849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:05.671998 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:05.672091 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:05.672210 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:05.673820 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:05.674665 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:05.676743 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:05.677240 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:05.677398 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:05.681816 (+ 4418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:05.681830 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:05.681833 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:17:05.681867 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:05.684515 (+ 2648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:17:05.685365 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:17:05.685373 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:17:05.685380 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:17:05.685517 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:17:05.685974 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:17:05.685978 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:17:05.685982 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:17:05.686271 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:05.686448 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:05.687185 (+ 737us) server_negotiation.cc:299] Negotiation successful
0924 16:17:05.687451 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":289,"threads_started":1}
I20260924 16:17:05.688900 15858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:05.669239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:46359 (local address 127.29.234.1:56689)
0924 16:17:05.669980 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0924 16:17:05.670024 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 16:17:05.671714 (+ 1690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:17:05.672456 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:17:05.672466 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:17:05.672967 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:17:05.673619 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:05.673639 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:05.674896 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:05.674901 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:05.676578 (+ 1677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:05.676594 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:05.677500 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:05.677509 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:05.677820 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:05.678560 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:17:05.678597 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:17:05.681539 (+ 2942us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:17:05.684702 (+ 3163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:17:05.684713 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:17:05.684739 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:17:05.685196 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:17:05.685637 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:17:05.685642 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:17:05.685646 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:17:05.685853 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:17:05.686409 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:17:05.686420 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:17:05.686790 (+ 370us) client_negotiation.cc:769] Sending connection context
0924 16:17:05.687078 (+ 288us) client_negotiation.cc:240] Negotiation successful
0924 16:17:05.687664 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":280,"threads_started":1}
I20260924 16:17:05.690726 15855 heartbeater.cc:347] Connected to a master server at 127.29.234.62:46359
I20260924 16:17:05.691084 15855 heartbeater.cc:464] Registering TS with master...
I20260924 16:17:05.691903 15855 heartbeater.cc:511] Master 127.29.234.62:46359 requested a full tablet report, sending...
I20260924 16:17:05.694258 15664 ts_manager.cc:194] Registered new tserver with Master: 902fcd0c4a5f4498b7d31bc1b28c62da (127.29.234.1:39455)
I20260924 16:17:05.695782 15664 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.234.1@KRBTEST.COM'} at 127.29.234.1:56689
WARNING: no policy specified for kudu/127.29.234.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab.
Entry for principal kudu/127.29.234.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab.
I20260924 16:17:05.732666 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.2:0
--local_ip_for_outbound_sockets=127.29.234.2
--webserver_interface=127.29.234.2
--webserver_port=0
--tserver_master_addrs=127.29.234.62:46359
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab
--principal=kudu/127.29.234.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:46873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:17:06.069089 15861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:17:06.069453 15861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:17:06.069561 15861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:17:06.076820 15861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:17:06.076922 15861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:17:06.077040 15861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.2
I20260924 16:17:06.087121 15861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:46873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab
--principal=kudu/127.29.234.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.234.2
--webserver_port=0
--tserver_master_addrs=127.29.234.62:46359
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.2
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:17:06.088431 15861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:17:06.089905 15861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:17:06.103703 15869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:06.103737 15867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:06.103703 15866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:17:06.104490 15861 server_base.cc:1034] running on GCE node
I20260924 16:17:06.105083 15861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:17:06.106225 15861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:17:06.107439 15861 hybrid_clock.cc:648] HybridClock initialized: now 1790266626107390 us; error 44 us; skew 500 ppm
I20260924 16:17:06.107781 15861 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:17:06 dist-test-slave-v5j9 krb5kdc[15619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266626, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:17:06.111299 15861 init.cc:377] Logged in from keytab as kudu/127.29.234.2@KRBTEST.COM (short username kudu)
I20260924 16:17:06.112830 15861 webserver.cc:460] Webserver started at http://127.29.234.2:39673/ using document root <none> and password file <none>
I20260924 16:17:06.113652 15861 fs_manager.cc:359] Metadata directory not provided
I20260924 16:17:06.113802 15861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:17:06.114142 15861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:17:06.117197 15861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/minicluster-data/ts-1/data/instance:
uuid: "aa297e5b7e8242419fcffc6ced0b9874"
format_stamp: "Formatted at 2026-09-24 16:17:06 on dist-test-slave-v5j9"
server_key: "2ab213c7b204ded8b6f4f85379f9edd3"
server_key_iv: "46cfcced69ab6533189f311e5c9a2fe1"
server_key_version: "encryptionkey@0"
I20260924 16:17:06.118017 15861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/minicluster-data/ts-1/wal/instance:
uuid: "aa297e5b7e8242419fcffc6ced0b9874"
format_stamp: "Formatted at 2026-09-24 16:17:06 on dist-test-slave-v5j9"
server_key: "2ab213c7b204ded8b6f4f85379f9edd3"
server_key_iv: "46cfcced69ab6533189f311e5c9a2fe1"
server_key_version: "encryptionkey@0"
I20260924 16:17:06.123684 15861 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 16:17:06.127285 15877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:17:06.128639 15861 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260924 16:17:06.128825 15861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/minicluster-data/ts-1/wal
uuid: "aa297e5b7e8242419fcffc6ced0b9874"
format_stamp: "Formatted at 2026-09-24 16:17:06 on dist-test-slave-v5j9"
server_key: "2ab213c7b204ded8b6f4f85379f9edd3"
server_key_iv: "46cfcced69ab6533189f311e5c9a2fe1"
server_key_version: "encryptionkey@0"
I20260924 16:17:06.129034 15861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:17:06.150928 15861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:17:06.155464 15861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:17:06.155814 15861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:17:06.161782 15861 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:17:06.163712 15861 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:17:06.163882 15861 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:17:06.164067 15861 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:17:06.164167 15861 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:17:06.200268 15861 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.2:40813
I20260924 16:17:06.200359 15993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.2:40813 every 8 connection(s)
I20260924 16:17:06.201741 15861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
I20260924 16:17:06.209578 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 15861
I20260924 16:17:06.209805 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/minicluster-data/ts-1/wal/instance
I20260924 16:17:06.210355 30632 external_mini_cluster.cc:1428] Setting key 009839ed982ef4f29cded27953d3c7f9
Sep 24 16:17:06 dist-test-slave-v5j9 krb5kdc[15619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266626, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.2@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:17:06.224143 15996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:06.204414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.2:47401 (local address 127.29.234.62:46359)
0924 16:17:06.204902 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:06.204906 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:06.205816 (+ 910us) server_negotiation.cc:407] Connection header received
0924 16:17:06.207740 (+ 1924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:06.207747 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:06.207838 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:06.207957 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:06.209557 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:06.210452 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:06.212625 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:06.213267 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:06.213441 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:06.217011 (+ 3570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:06.217020 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:06.217028 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:17:06.217077 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:06.220763 (+ 3686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:17:06.221576 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:17:06.221583 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:17:06.221588 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:17:06.221759 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:17:06.222185 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:17:06.222191 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:17:06.222194 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:17:06.222592 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:06.222745 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:06.223697 (+ 952us) server_negotiation.cc:299] Negotiation successful
0924 16:17:06.223879 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":268,"threads_started":1}
I20260924 16:17:06.225821 15997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:06.204833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:46359 (local address 127.29.234.2:47401)
0924 16:17:06.205638 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0924 16:17:06.205679 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 16:17:06.207471 (+ 1792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:17:06.208206 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:17:06.208215 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:17:06.208724 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:17:06.209326 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:06.209402 (+ 76us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:06.210616 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:06.210622 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:06.212455 (+ 1833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:06.212471 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:06.213547 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:06.213552 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:06.213772 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:06.214480 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:17:06.214509 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:17:06.216746 (+ 2237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:17:06.220924 (+ 4178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:17:06.220931 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:17:06.220946 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:17:06.221457 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:17:06.221872 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:17:06.221876 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:17:06.221878 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:17:06.222062 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:17:06.222761 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:17:06.222771 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:17:06.223265 (+ 494us) client_negotiation.cc:769] Sending connection context
0924 16:17:06.223592 (+ 327us) client_negotiation.cc:240] Negotiation successful
0924 16:17:06.224220 (+ 628us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":294,"threads_started":1}
I20260924 16:17:06.227466 15994 heartbeater.cc:347] Connected to a master server at 127.29.234.62:46359
I20260924 16:17:06.227831 15994 heartbeater.cc:464] Registering TS with master...
I20260924 16:17:06.228736 15994 heartbeater.cc:511] Master 127.29.234.62:46359 requested a full tablet report, sending...
I20260924 16:17:06.230547 15664 ts_manager.cc:194] Registered new tserver with Master: aa297e5b7e8242419fcffc6ced0b9874 (127.29.234.2:40813)
I20260924 16:17:06.231832 15664 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.234.2@KRBTEST.COM'} at 127.29.234.2:47401
WARNING: no policy specified for kudu/127.29.234.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab.
Entry for principal kudu/127.29.234.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab.
I20260924 16:17:06.271065 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.3:0
--local_ip_for_outbound_sockets=127.29.234.3
--webserver_interface=127.29.234.3
--webserver_port=0
--tserver_master_addrs=127.29.234.62:46359
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab
--principal=kudu/127.29.234.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:46873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:17:06.606879 16000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:17:06.607213 16000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:17:06.607389 16000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:17:06.614790 16000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:17:06.614936 16000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:17:06.615131 16000 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.3
I20260924 16:17:06.625551 16000 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:46873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab
--principal=kudu/127.29.234.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.234.3
--webserver_port=0
--tserver_master_addrs=127.29.234.62:46359
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.3
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:17:06.626963 16000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:17:06.628353 16000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:17:06.642344 16005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:06.642446 16008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:06.642398 16006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:17:06.643425 16000 server_base.cc:1034] running on GCE node
I20260924 16:17:06.644093 16000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:17:06.645068 16000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:17:06.646330 16000 hybrid_clock.cc:648] HybridClock initialized: now 1790266626646247 us; error 75 us; skew 500 ppm
I20260924 16:17:06.646673 16000 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:17:06 dist-test-slave-v5j9 krb5kdc[15619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266626, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:17:06.650089 16000 init.cc:377] Logged in from keytab as kudu/127.29.234.3@KRBTEST.COM (short username kudu)
I20260924 16:17:06.651623 16000 webserver.cc:460] Webserver started at http://127.29.234.3:40793/ using document root <none> and password file <none>
I20260924 16:17:06.652561 16000 fs_manager.cc:359] Metadata directory not provided
I20260924 16:17:06.652715 16000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:17:06.653046 16000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:17:06.656075 16000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/minicluster-data/ts-2/data/instance:
uuid: "d32bf07da97342eb980c0ca29c7e8877"
format_stamp: "Formatted at 2026-09-24 16:17:06 on dist-test-slave-v5j9"
server_key: "1cc3cf8323b91cda7eeb44eefc94c59a"
server_key_iv: "e660ca88ff91619c6eccd9cd9bf6e4be"
server_key_version: "encryptionkey@0"
I20260924 16:17:06.656872 16000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/minicluster-data/ts-2/wal/instance:
uuid: "d32bf07da97342eb980c0ca29c7e8877"
format_stamp: "Formatted at 2026-09-24 16:17:06 on dist-test-slave-v5j9"
server_key: "1cc3cf8323b91cda7eeb44eefc94c59a"
server_key_iv: "e660ca88ff91619c6eccd9cd9bf6e4be"
server_key_version: "encryptionkey@0"
I20260924 16:17:06.662197 16000 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 16:17:06.665864 16015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:17:06.667323 16000 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260924 16:17:06.667500 16000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/minicluster-data/ts-2/wal
uuid: "d32bf07da97342eb980c0ca29c7e8877"
format_stamp: "Formatted at 2026-09-24 16:17:06 on dist-test-slave-v5j9"
server_key: "1cc3cf8323b91cda7eeb44eefc94c59a"
server_key_iv: "e660ca88ff91619c6eccd9cd9bf6e4be"
server_key_version: "encryptionkey@0"
I20260924 16:17:06.667709 16000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:17:06.681195 16000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:17:06.685113 16000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:17:06.685483 16000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:17:06.686452 16000 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:17:06.687758 16000 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:17:06.687862 16000 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:17:06.688010 16000 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:17:06.688088 16000 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:17:06.699012 15855 heartbeater.cc:503] Master 127.29.234.62:46359 was elected leader, sending a full tablet report...
I20260924 16:17:06.721087 16000 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.3:34041
I20260924 16:17:06.721143 16129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.3:34041 every 8 connection(s)
I20260924 16:17:06.722534 16000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
I20260924 16:17:06.724464 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 16000
I20260924 16:17:06.724658 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/minicluster-data/ts-2/wal/instance
I20260924 16:17:06.725250 30632 external_mini_cluster.cc:1428] Setting key 36e9e5a9099336f054c16ec4d6beefb0
Sep 24 16:17:06 dist-test-slave-v5j9 krb5kdc[15619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266626, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.3@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:17:06.743484 16132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:06.725082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.3:44383 (local address 127.29.234.62:46359)
0924 16:17:06.725615 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:06.725623 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:06.726891 (+ 1268us) server_negotiation.cc:407] Connection header received
0924 16:17:06.728695 (+ 1804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:06.728700 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:06.728802 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:06.728914 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:06.730526 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:06.731299 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:06.733008 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:06.733466 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:06.733594 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:06.737222 (+ 3628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:06.737226 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:06.737230 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:17:06.737265 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:06.740330 (+ 3065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:17:06.741275 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:17:06.741279 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:17:06.741281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:17:06.741460 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:17:06.741871 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:17:06.741874 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:17:06.741876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:17:06.742168 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:06.742275 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:06.743029 (+ 754us) server_negotiation.cc:299] Negotiation successful
0924 16:17:06.743215 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":244,"threads_started":1}
I20260924 16:17:06.744480 16133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:06.725730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:46359 (local address 127.29.234.3:44383)
0924 16:17:06.726596 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0924 16:17:06.726645 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 16:17:06.728432 (+ 1787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:17:06.729163 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:17:06.729172 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:17:06.729771 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:17:06.730359 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:06.730377 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:06.731461 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:06.731465 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:06.732844 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:06.732857 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:06.733702 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:06.733709 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:06.734000 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:06.734696 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:17:06.734718 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:17:06.737001 (+ 2283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:17:06.740501 (+ 3500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:17:06.740512 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:17:06.740533 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:17:06.741169 (+ 636us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:17:06.741574 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:17:06.741578 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:17:06.741580 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:17:06.741775 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:17:06.742281 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:17:06.742287 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:17:06.742651 (+ 364us) client_negotiation.cc:769] Sending connection context
0924 16:17:06.742898 (+ 247us) client_negotiation.cc:240] Negotiation successful
0924 16:17:06.743331 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":358,"threads_started":1}
I20260924 16:17:06.745898 16130 heartbeater.cc:347] Connected to a master server at 127.29.234.62:46359
I20260924 16:17:06.746237 16130 heartbeater.cc:464] Registering TS with master...
I20260924 16:17:06.747038 16130 heartbeater.cc:511] Master 127.29.234.62:46359 requested a full tablet report, sending...
I20260924 16:17:06.748471 15664 ts_manager.cc:194] Registered new tserver with Master: d32bf07da97342eb980c0ca29c7e8877 (127.29.234.3:34041)
I20260924 16:17:06.749297 15664 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.234.3@KRBTEST.COM'} at 127.29.234.3:44383
I20260924 16:17:06.753299 30632 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:17:06.770290 16132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:06.757876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48516 (local address 127.29.234.62:46359)
0924 16:17:06.758033 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:06.758038 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:06.758554 (+ 516us) server_negotiation.cc:407] Connection header received
0924 16:17:06.758699 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:06.758703 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:06.758764 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:06.758843 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:06.760232 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:06.760942 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:06.762267 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:06.762646 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:06.762744 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:06.765106 (+ 2362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:06.765111 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:06.765114 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:17:06.765146 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:06.767847 (+ 2701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:17:06.768493 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:17:06.768497 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:17:06.768499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:17:06.768615 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:17:06.768989 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:17:06.768992 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:17:06.768994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:17:06.769303 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:06.769616 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:06.769889 (+ 273us) server_negotiation.cc:299] Negotiation successful
0924 16:17:06.770059 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 16:17:06.774326 15664 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48516:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:17:06.781558 15664 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:17:06.805758 16144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:06.794008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:39455 (local address 127.0.0.1:59260)
0924 16:17:06.795528 (+ 1520us) negotiation.cc:107] Waiting for socket to connect
0924 16:17:06.795575 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 16:17:06.795740 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:17:06.797238 (+ 1498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:17:06.797246 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:17:06.797293 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:17:06.797767 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:06.797780 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:06.799853 (+ 2073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:06.799858 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:06.802575 (+ 2717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:06.802588 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:06.804621 (+ 2033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:06.804628 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:06.804816 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:06.804905 (+ 89us) client_negotiation.cc:769] Sending connection context
0924 16:17:06.805044 (+ 139us) client_negotiation.cc:240] Negotiation successful
0924 16:17:06.805237 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1277,"mutex_wait_us":82,"thread_start_us":1089,"threads_started":1}
I20260924 16:17:06.805775 16143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:06.794022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59260 (local address 127.29.234.1:39455)
0924 16:17:06.796611 (+ 2589us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:06.796618 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:06.796640 (+ 22us) server_negotiation.cc:407] Connection header received
0924 16:17:06.796723 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:06.796729 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:06.796923 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:06.797968 (+ 1045us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:17:06.798228 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:06.799704 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:06.802733 (+ 3029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:06.804250 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:06.804400 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:06.804658 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:06.805243 (+ 585us) server_negotiation.cc:299] Negotiation successful
0924 16:17:06.805449 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2451,"thread_start_us":2336,"threads_started":1}
I20260924 16:17:06.806799 16147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:06.797085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.3:34041 (local address 127.0.0.1:45438)
0924 16:17:06.799033 (+ 1948us) negotiation.cc:107] Waiting for socket to connect
0924 16:17:06.799050 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:17:06.799193 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:17:06.799750 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:17:06.799757 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:17:06.799789 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:17:06.800287 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:06.800300 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:06.802325 (+ 2025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:06.802330 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:06.803826 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:06.803838 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:06.805909 (+ 2071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:06.805918 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:06.806115 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:06.806158 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 16:17:06.806251 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 16:17:06.806386 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1813,"mutex_wait_us":27,"thread_start_us":1686,"threads_started":1}
I20260924 16:17:06.807057 16146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:06.794555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38740 (local address 127.29.234.2:40813)
0924 16:17:06.796209 (+ 1654us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:06.796215 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:06.797271 (+ 1056us) server_negotiation.cc:407] Connection header received
0924 16:17:06.797419 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:06.797427 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:06.797671 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:06.797877 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:17:06.798878 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:06.800273 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:06.802852 (+ 2579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:06.804264 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:06.804403 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:06.804612 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:06.806604 (+ 1992us) server_negotiation.cc:299] Negotiation successful
0924 16:17:06.806779 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1505,"thread_start_us":1417,"threads_started":1}
I20260924 16:17:06.807542 16145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:06.794141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.2:40813 (local address 127.0.0.1:38740)
0924 16:17:06.797000 (+ 2859us) negotiation.cc:107] Waiting for socket to connect
0924 16:17:06.797019 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 16:17:06.797143 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:17:06.798137 (+ 994us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:17:06.798144 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:17:06.798177 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:17:06.798624 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:06.798642 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:06.800415 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:06.800424 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:06.802688 (+ 2264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:06.802702 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:06.806204 (+ 3502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:06.806212 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:06.806396 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:06.806424 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 16:17:06.806515 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 16:17:06.806636 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2721,"mutex_wait_us":34,"thread_start_us":2577,"threads_started":1}
I20260924 16:17:06.807870 16148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:06.797623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45438 (local address 127.29.234.3:34041)
0924 16:17:06.798626 (+ 1003us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:06.798635 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:06.799161 (+ 526us) server_negotiation.cc:407] Connection header received
0924 16:17:06.799327 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:06.799335 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:06.799547 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:06.799740 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:17:06.800498 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:06.802171 (+ 1673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:06.803984 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:06.805392 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:06.805514 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:06.805743 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:06.806913 (+ 1170us) server_negotiation.cc:299] Negotiation successful
0924 16:17:06.807090 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":827,"thread_start_us":678,"threads_started":1}
I20260924 16:17:06.810299 15789 tablet_service.cc:1467] Processing CreateTablet for tablet 33c90ae4cdf44bd8a7b6bfd7bb242279 (DEFAULT_TABLE table=test-table [id=fba85f7caeff448bb45cd2996094387d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:17:06.810909 16065 tablet_service.cc:1467] Processing CreateTablet for tablet 33c90ae4cdf44bd8a7b6bfd7bb242279 (DEFAULT_TABLE table=test-table [id=fba85f7caeff448bb45cd2996094387d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:17:06.811144 15927 tablet_service.cc:1467] Processing CreateTablet for tablet 33c90ae4cdf44bd8a7b6bfd7bb242279 (DEFAULT_TABLE table=test-table [id=fba85f7caeff448bb45cd2996094387d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:17:06.811853 15789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33c90ae4cdf44bd8a7b6bfd7bb242279. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:17:06.812140 16065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33c90ae4cdf44bd8a7b6bfd7bb242279. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:17:06.812450 15927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33c90ae4cdf44bd8a7b6bfd7bb242279. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:17:06.823832 16149 tablet_bootstrap.cc:492] T 33c90ae4cdf44bd8a7b6bfd7bb242279 P d32bf07da97342eb980c0ca29c7e8877: Bootstrap starting.
I20260924 16:17:06.827158 16150 tablet_bootstrap.cc:492] T 33c90ae4cdf44bd8a7b6bfd7bb242279 P aa297e5b7e8242419fcffc6ced0b9874: Bootstrap starting.
I20260924 16:17:06.827956 16151 tablet_bootstrap.cc:492] T 33c90ae4cdf44bd8a7b6bfd7bb242279 P 902fcd0c4a5f4498b7d31bc1b28c62da: Bootstrap starting.
I20260924 16:17:06.828806 16149 tablet_bootstrap.cc:654] T 33c90ae4cdf44bd8a7b6bfd7bb242279 P d32bf07da97342eb980c0ca29c7e8877: Neither blocks nor log segments found. Creating new log.
I20260924 16:17:06.830222 16149 log.cc:824] T 33c90ae4cdf44bd8a7b6bfd7bb242279 P d32bf07da97342eb980c0ca29c7e8877: Log is configured to *not* fsync() on all Append() calls
I20260924 16:17:06.831568 16151 tablet_bootstrap.cc:654] T 33c90ae4cdf44bd8a7b6bfd7bb242279 P 902fcd0c4a5f4498b7d31bc1b28c62da: Neither blocks nor log segments found. Creating new log.
I20260924 16:17:06.832304 16150 tablet_bootstrap.cc:654] T 33c90ae4cdf44bd8a7b6bfd7bb242279 P aa297e5b7e8242419fcffc6ced0b9874: Neither blocks nor log segments found. Creating new log.
I20260924 16:17:06.832852 16151 log.cc:824] T 33c90ae4cdf44bd8a7b6bfd7bb242279 P 902fcd0c4a5f4498b7d31bc1b28c62da: Log is configured to *not* fsync() on all Append() calls
I20260924 16:17:06.833421 16150 log.cc:824] T 33c90ae4cdf44bd8a7b6bfd7bb242279 P aa297e5b7e8242419fcffc6ced0b9874: Log is configured to *not* fsync() on all Append() calls
I20260924 16:17:06.833618 16149 tablet_bootstrap.cc:492] T 33c90ae4cdf44bd8a7b6bfd7bb242279 P d32bf07da97342eb980c0ca29c7e8877: No bootstrap required, opened a new log
I20260924 16:17:06.833909 16149 ts_tablet_manager.cc:1397] T 33c90ae4cdf44bd8a7b6bfd7bb242279 P d32bf07da97342eb980c0ca29c7e8877: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260924 16:17:06.836117 16151 tablet_bootstrap.cc:492] T 33c90ae4cdf44bd8a7b6bfd7bb242279 P 902fcd0c4a5f4498b7d31bc1b28c62da: No bootstrap required, opened a new log
I20260924 16:17:06.836453 16151 ts_tablet_manager.cc:1397] T 33c90ae4cdf44bd8a7b6bfd7bb242279 P 902fcd0c4a5f4498b7d31bc1b28c62da: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.002s
I20260924 16:17:06.839402 16150 tablet_bootstrap.cc:492] T 33c90ae4cdf44bd8a7b6bfd7bb242279 P aa297e5b7e8242419fcffc6ced0b9874: No bootstrap required, opened a new log
I20260924 16:17:06.839668 16150 ts_tablet_manager.cc:1397] T 33c90ae4cdf44bd8a7b6bfd7bb242279 P aa297e5b7e8242419fcffc6ced0b9874: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.004s
I20260924 16:17:06.845853 16149 raft_consensus.cc:348] T 33c90ae4cdf44bd8a7b6bfd7bb242279 P d32bf07da97342eb980c0ca29c7e8877 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "902fcd0c4a5f4498b7d31bc1b28c62da" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 39455 } } peers { permanent_uuid: "aa297e5b7e8242419fcffc6ced0b9874" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 40813 } } peers { permanent_uuid: "d32bf07da97342eb980c0ca29c7e8877" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 34041 } }
I20260924 16:17:06.846393 16149 raft_consensus.cc:374] T 33c90ae4cdf44bd8a7b6bfd7bb242279 P d32bf07da97342eb980c0ca29c7e8877 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:17:06.846522 16149 raft_consensus.cc:729] T 33c90ae4cdf44bd8a7b6bfd7bb242279 P d32bf07da97342eb980c0ca29c7e8877 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d32bf07da97342eb980c0ca29c7e8877, State: Initialized, Role: FOLLOWER
I20260924 16:17:06.846771 16150 raft_consensus.cc:348] T 33c90ae4cdf44bd8a7b6bfd7bb242279 P aa297e5b7e8242419fcffc6ced0b9874 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "902fcd0c4a5f4498b7d31bc1b28c62da" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 39455 } } peers { permanent_uuid: "aa297e5b7e8242419fcffc6ced0b9874" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 40813 } } peers { permanent_uuid: "d32bf07da97342eb980c0ca29c7e8877" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 34041 } }
I20260924 16:17:06.847259 16150 raft_consensus.cc:374] T 33c90ae4cdf44bd8a7b6bfd7bb242279 P aa297e5b7e8242419fcffc6ced0b9874 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:17:06.847390 16150 raft_consensus.cc:729] T 33c90ae4cdf44bd8a7b6bfd7bb242279 P aa297e5b7e8242419fcffc6ced0b9874 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa297e5b7e8242419fcffc6ced0b9874, State: Initialized, Role: FOLLOWER
I20260924 16:17:06.847213 16149 consensus_queue.cc:260] T 33c90ae4cdf44bd8a7b6bfd7bb242279 P d32bf07da97342eb980c0ca29c7e8877 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "902fcd0c4a5f4498b7d31bc1b28c62da" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 39455 } } peers { permanent_uuid: "aa297e5b7e8242419fcffc6ced0b9874" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 40813 } } peers { permanent_uuid: "d32bf07da97342eb980c0ca29c7e8877" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 34041 } }
I20260924 16:17:06.847406 16151 raft_consensus.cc:348] T 33c90ae4cdf44bd8a7b6bfd7bb242279 P 902fcd0c4a5f4498b7d31bc1b28c62da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "902fcd0c4a5f4498b7d31bc1b28c62da" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 39455 } } peers { permanent_uuid: "aa297e5b7e8242419fcffc6ced0b9874" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 40813 } } peers { permanent_uuid: "d32bf07da97342eb980c0ca29c7e8877" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 34041 } }
I20260924 16:17:06.847908 16151 raft_consensus.cc:374] T 33c90ae4cdf44bd8a7b6bfd7bb242279 P 902fcd0c4a5f4498b7d31bc1b28c62da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:17:06.848104 16150 consensus_queue.cc:260] T 33c90ae4cdf44bd8a7b6bfd7bb242279 P aa297e5b7e8242419fcffc6ced0b9874 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "902fcd0c4a5f4498b7d31bc1b28c62da" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 39455 } } peers { permanent_uuid: "aa297e5b7e8242419fcffc6ced0b9874" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 40813 } } peers { permanent_uuid: "d32bf07da97342eb980c0ca29c7e8877" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 34041 } }
I20260924 16:17:06.849090 16155 raft_consensus.cc:482] T 33c90ae4cdf44bd8a7b6bfd7bb242279 P d32bf07da97342eb980c0ca29c7e8877 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:17:06.849207 16130 heartbeater.cc:503] Master 127.29.234.62:46359 was elected leader, sending a full tablet report...
I20260924 16:17:06.849532 16155 raft_consensus.cc:504] T 33c90ae4cdf44bd8a7b6bfd7bb242279 P d32bf07da97342eb980c0ca29c7e8877 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "902fcd0c4a5f4498b7d31bc1b28c62da" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 39455 } } peers { permanent_uuid: "aa297e5b7e8242419fcffc6ced0b9874" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 40813 } } peers { permanent_uuid: "d32bf07da97342eb980c0ca29c7e8877" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 34041 } }
I20260924 16:17:06.852087 16155 leader_election.cc:288] T 33c90ae4cdf44bd8a7b6bfd7bb242279 P d32bf07da97342eb980c0ca29c7e8877 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 902fcd0c4a5f4498b7d31bc1b28c62da (127.29.234.1:39455), aa297e5b7e8242419fcffc6ced0b9874 (127.29.234.2:40813)
I20260924 16:17:06.854054 16149 ts_tablet_manager.cc:1428] T 33c90ae4cdf44bd8a7b6bfd7bb242279 P d32bf07da97342eb980c0ca29c7e8877: Time spent starting tablet: real 0.020s user 0.014s sys 0.001s
I20260924 16:17:06.854816 16151 raft_consensus.cc:729] T 33c90ae4cdf44bd8a7b6bfd7bb242279 P 902fcd0c4a5f4498b7d31bc1b28c62da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 902fcd0c4a5f4498b7d31bc1b28c62da, State: Initialized, Role: FOLLOWER
I20260924 16:17:06.855948 15994 heartbeater.cc:503] Master 127.29.234.62:46359 was elected leader, sending a full tablet report...
I20260924 16:17:06.856220 16151 consensus_queue.cc:260] T 33c90ae4cdf44bd8a7b6bfd7bb242279 P 902fcd0c4a5f4498b7d31bc1b28c62da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "902fcd0c4a5f4498b7d31bc1b28c62da" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 39455 } } peers { permanent_uuid: "aa297e5b7e8242419fcffc6ced0b9874" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 40813 } } peers { permanent_uuid: "d32bf07da97342eb980c0ca29c7e8877" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 34041 } }
I20260924 16:17:06.861792 16133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:06.852344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:39455 (local address 127.29.234.3:33645)
0924 16:17:06.852834 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0924 16:17:06.852856 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 16:17:06.852965 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:17:06.854173 (+ 1208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:17:06.854181 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:17:06.854221 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:17:06.854474 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:06.854489 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:06.856160 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:06.856167 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:06.858590 (+ 2423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:06.858609 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:06.860706 (+ 2097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:06.860715 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:06.860990 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:06.861043 (+ 53us) client_negotiation.cc:769] Sending connection context
0924 16:17:06.861142 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 16:17:06.861419 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344}
I20260924 16:17:06.862895 16143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:06.853324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.3:33645 (local address 127.29.234.1:39455)
0924 16:17:06.853592 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:06.853601 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:06.853628 (+ 27us) server_negotiation.cc:407] Connection header received
0924 16:17:06.853718 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:06.853727 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:06.853823 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:06.853941 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:17:06.854675 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:06.855937 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:06.858826 (+ 2889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:06.860312 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:06.860492 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:06.860602 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:06.862375 (+ 1773us) server_negotiation.cc:299] Negotiation successful
0924 16:17:06.862574 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260924 16:17:06.863286 16146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:06.855154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.3:39513 (local address 127.29.234.2:40813)
0924 16:17:06.855574 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:06.855583 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:06.856261 (+ 678us) server_negotiation.cc:407] Connection header received
0924 16:17:06.856524 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:06.856530 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:06.856590 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:06.856652 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:17:06.857522 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:06.858529 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:06.860967 (+ 2438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:06.862178 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:06.862329 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:06.862407 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:06.862903 (+ 496us) server_negotiation.cc:299] Negotiation successful
0924 16:17:06.863033 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242}
I20260924 16:17:06.863379 16158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:06.854938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.2:40813 (local address 127.29.234.3:39513)
0924 16:17:06.856165 (+ 1227us) negotiation.cc:107] Waiting for socket to connect
0924 16:17:06.856183 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 16:17:06.856368 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:17:06.856842 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:17:06.856846 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:17:06.856869 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:17:06.857305 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:06.857322 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:06.858732 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:06.858737 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:06.860773 (+ 2036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:06.860787 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:06.862591 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:06.862600 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:06.862736 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:06.862758 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 16:17:06.862876 (+ 118us) client_negotiation.cc:240] Negotiation successful
0924 16:17:06.863006 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1099,"thread_start_us":4213,"threads_started":1}
I20260924 16:17:06.864465 15947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33c90ae4cdf44bd8a7b6bfd7bb242279" candidate_uuid: "d32bf07da97342eb980c0ca29c7e8877" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa297e5b7e8242419fcffc6ced0b9874" is_pre_election: true
I20260924 16:17:06.864460 15809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33c90ae4cdf44bd8a7b6bfd7bb242279" candidate_uuid: "d32bf07da97342eb980c0ca29c7e8877" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "902fcd0c4a5f4498b7d31bc1b28c62da" is_pre_election: true
I20260924 16:17:06.865064 15947 raft_consensus.cc:2454] T 33c90ae4cdf44bd8a7b6bfd7bb242279 P aa297e5b7e8242419fcffc6ced0b9874 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d32bf07da97342eb980c0ca29c7e8877 in term 0.
I20260924 16:17:06.865064 15809 raft_consensus.cc:2454] T 33c90ae4cdf44bd8a7b6bfd7bb242279 P 902fcd0c4a5f4498b7d31bc1b28c62da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d32bf07da97342eb980c0ca29c7e8877 in term 0.
I20260924 16:17:06.865739 16150 ts_tablet_manager.cc:1428] T 33c90ae4cdf44bd8a7b6bfd7bb242279 P aa297e5b7e8242419fcffc6ced0b9874: Time spent starting tablet: real 0.026s user 0.012s sys 0.004s
I20260924 16:17:06.865932 16016 leader_election.cc:302] T 33c90ae4cdf44bd8a7b6bfd7bb242279 P d32bf07da97342eb980c0ca29c7e8877 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 902fcd0c4a5f4498b7d31bc1b28c62da, aa297e5b7e8242419fcffc6ced0b9874, d32bf07da97342eb980c0ca29c7e8877; no voters:
I20260924 16:17:06.866793 16156 raft_consensus.cc:2781] T 33c90ae4cdf44bd8a7b6bfd7bb242279 P d32bf07da97342eb980c0ca29c7e8877 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:17:06.867010 16156 raft_consensus.cc:482] T 33c90ae4cdf44bd8a7b6bfd7bb242279 P d32bf07da97342eb980c0ca29c7e8877 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:17:06.867224 16156 raft_consensus.cc:3037] T 33c90ae4cdf44bd8a7b6bfd7bb242279 P d32bf07da97342eb980c0ca29c7e8877 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:17:06.867501 16151 ts_tablet_manager.cc:1428] T 33c90ae4cdf44bd8a7b6bfd7bb242279 P 902fcd0c4a5f4498b7d31bc1b28c62da: Time spent starting tablet: real 0.031s user 0.011s sys 0.007s
I20260924 16:17:06.870333 16156 raft_consensus.cc:504] T 33c90ae4cdf44bd8a7b6bfd7bb242279 P d32bf07da97342eb980c0ca29c7e8877 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "902fcd0c4a5f4498b7d31bc1b28c62da" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 39455 } } peers { permanent_uuid: "aa297e5b7e8242419fcffc6ced0b9874" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 40813 } } peers { permanent_uuid: "d32bf07da97342eb980c0ca29c7e8877" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 34041 } }
I20260924 16:17:06.871325 16156 leader_election.cc:288] T 33c90ae4cdf44bd8a7b6bfd7bb242279 P d32bf07da97342eb980c0ca29c7e8877 [CANDIDATE]: Term 1 election: Requested vote from peers 902fcd0c4a5f4498b7d31bc1b28c62da (127.29.234.1:39455), aa297e5b7e8242419fcffc6ced0b9874 (127.29.234.2:40813)
I20260924 16:17:06.871611 15947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33c90ae4cdf44bd8a7b6bfd7bb242279" candidate_uuid: "d32bf07da97342eb980c0ca29c7e8877" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa297e5b7e8242419fcffc6ced0b9874"
I20260924 16:17:06.871604 15809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33c90ae4cdf44bd8a7b6bfd7bb242279" candidate_uuid: "d32bf07da97342eb980c0ca29c7e8877" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "902fcd0c4a5f4498b7d31bc1b28c62da"
I20260924 16:17:06.871829 15947 raft_consensus.cc:3037] T 33c90ae4cdf44bd8a7b6bfd7bb242279 P aa297e5b7e8242419fcffc6ced0b9874 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:17:06.871832 15809 raft_consensus.cc:3037] T 33c90ae4cdf44bd8a7b6bfd7bb242279 P 902fcd0c4a5f4498b7d31bc1b28c62da [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:17:06.875074 15809 raft_consensus.cc:2454] T 33c90ae4cdf44bd8a7b6bfd7bb242279 P 902fcd0c4a5f4498b7d31bc1b28c62da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d32bf07da97342eb980c0ca29c7e8877 in term 1.
I20260924 16:17:06.875074 15947 raft_consensus.cc:2454] T 33c90ae4cdf44bd8a7b6bfd7bb242279 P aa297e5b7e8242419fcffc6ced0b9874 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d32bf07da97342eb980c0ca29c7e8877 in term 1.
I20260924 16:17:06.875511 16016 leader_election.cc:302] T 33c90ae4cdf44bd8a7b6bfd7bb242279 P d32bf07da97342eb980c0ca29c7e8877 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa297e5b7e8242419fcffc6ced0b9874, d32bf07da97342eb980c0ca29c7e8877; no voters:
I20260924 16:17:06.875838 16156 raft_consensus.cc:2781] T 33c90ae4cdf44bd8a7b6bfd7bb242279 P d32bf07da97342eb980c0ca29c7e8877 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:17:06.876061 16156 raft_consensus.cc:686] T 33c90ae4cdf44bd8a7b6bfd7bb242279 P d32bf07da97342eb980c0ca29c7e8877 [term 1 LEADER]: Becoming Leader. State: Replica: d32bf07da97342eb980c0ca29c7e8877, State: Running, Role: LEADER
I20260924 16:17:06.876659 16156 consensus_queue.cc:237] T 33c90ae4cdf44bd8a7b6bfd7bb242279 P d32bf07da97342eb980c0ca29c7e8877 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "902fcd0c4a5f4498b7d31bc1b28c62da" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 39455 } } peers { permanent_uuid: "aa297e5b7e8242419fcffc6ced0b9874" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 40813 } } peers { permanent_uuid: "d32bf07da97342eb980c0ca29c7e8877" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 34041 } }
I20260924 16:17:06.881450 15664 catalog_manager.cc:5495] T 33c90ae4cdf44bd8a7b6bfd7bb242279 P d32bf07da97342eb980c0ca29c7e8877 reported cstate change: term changed from 0 to 1, leader changed from <none> to d32bf07da97342eb980c0ca29c7e8877 (127.29.234.3). New cstate: current_term: 1 leader_uuid: "d32bf07da97342eb980c0ca29c7e8877" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "902fcd0c4a5f4498b7d31bc1b28c62da" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 39455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa297e5b7e8242419fcffc6ced0b9874" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 40813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d32bf07da97342eb980c0ca29c7e8877" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 34041 } health_report { overall_health: HEALTHY } } }
I20260924 16:17:06.908861 16148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:06.903146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45446 (local address 127.29.234.3:34041)
0924 16:17:06.903352 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:06.903358 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:06.903378 (+ 20us) server_negotiation.cc:407] Connection header received
0924 16:17:06.903521 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:06.903525 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:06.903603 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:06.903754 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:17:06.904138 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:06.905083 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:06.906633 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:06.907051 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:06.907188 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:06.907727 (+ 539us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:17:06.908264 (+ 537us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:17:06.908444 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:06.908495 (+ 51us) server_negotiation.cc:299] Negotiation successful
0924 16:17:06.908580 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 16:17:06.917467 15947 raft_consensus.cc:1261] T 33c90ae4cdf44bd8a7b6bfd7bb242279 P aa297e5b7e8242419fcffc6ced0b9874 [term 1 FOLLOWER]: Refusing update from remote peer d32bf07da97342eb980c0ca29c7e8877: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:17:06.917469 15809 raft_consensus.cc:1261] T 33c90ae4cdf44bd8a7b6bfd7bb242279 P 902fcd0c4a5f4498b7d31bc1b28c62da [term 1 FOLLOWER]: Refusing update from remote peer d32bf07da97342eb980c0ca29c7e8877: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260924 16:17:06.918594 15856 tablet.cc:2369] T 33c90ae4cdf44bd8a7b6bfd7bb242279 P 902fcd0c4a5f4498b7d31bc1b28c62da: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:17:06.918516 16156 consensus_queue.cc:1035] T 33c90ae4cdf44bd8a7b6bfd7bb242279 P d32bf07da97342eb980c0ca29c7e8877 [LEADER]: Connected to new peer: Peer: permanent_uuid: "902fcd0c4a5f4498b7d31bc1b28c62da" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 39455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:17:06.918834 16155 consensus_queue.cc:1035] T 33c90ae4cdf44bd8a7b6bfd7bb242279 P d32bf07da97342eb980c0ca29c7e8877 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa297e5b7e8242419fcffc6ced0b9874" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 40813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:17:06.929950 16162 mvcc.cc:205] Tried to move back new op lower bound from 7332932103843262464 to 7332932103691055104. Current Snapshot: MvccSnapshot[applied={T|T < 7332932103843262464}]
I20260924 16:17:06.944470 15664 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48516:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:17:06.944774 15664 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48516:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:17:06.948753 15664 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2b3ed99e62f840fdb722c5df44e02389: Sending DeleteTablet for 3 replicas of tablet 33c90ae4cdf44bd8a7b6bfd7bb242279
I20260924 16:17:06.950014 15789 tablet_service.cc:1514] Processing DeleteTablet for tablet 33c90ae4cdf44bd8a7b6bfd7bb242279 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:17:06 UTC) from {username='kudu', principal='kudu/127.29.234.62@KRBTEST.COM'} at 127.0.0.1:59260
I20260924 16:17:06.950028 16065 tablet_service.cc:1514] Processing DeleteTablet for tablet 33c90ae4cdf44bd8a7b6bfd7bb242279 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:17:06 UTC) from {username='kudu', principal='kudu/127.29.234.62@KRBTEST.COM'} at 127.0.0.1:45438
I20260924 16:17:06.950484 15927 tablet_service.cc:1514] Processing DeleteTablet for tablet 33c90ae4cdf44bd8a7b6bfd7bb242279 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:17:06 UTC) from {username='kudu', principal='kudu/127.29.234.62@KRBTEST.COM'} at 127.0.0.1:38740
I20260924 16:17:06.954782 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 15726
I20260924 16:17:06.963047 16174 tablet_replica.cc:330] T 33c90ae4cdf44bd8a7b6bfd7bb242279 P d32bf07da97342eb980c0ca29c7e8877: stopping tablet replica
I20260924 16:17:06.963852 16174 raft_consensus.cc:2229] T 33c90ae4cdf44bd8a7b6bfd7bb242279 P d32bf07da97342eb980c0ca29c7e8877 [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:17:06.964910 16174 raft_consensus.cc:2258] T 33c90ae4cdf44bd8a7b6bfd7bb242279 P d32bf07da97342eb980c0ca29c7e8877 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:17:06.967837 16174 ts_tablet_manager.cc:1905] T 33c90ae4cdf44bd8a7b6bfd7bb242279 P d32bf07da97342eb980c0ca29c7e8877: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:17:06.968586 16173 tablet_replica.cc:330] T 33c90ae4cdf44bd8a7b6bfd7bb242279 P 902fcd0c4a5f4498b7d31bc1b28c62da: stopping tablet replica
I20260924 16:17:06.969261 16173 raft_consensus.cc:2229] T 33c90ae4cdf44bd8a7b6bfd7bb242279 P 902fcd0c4a5f4498b7d31bc1b28c62da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:17:06.970058 16173 raft_consensus.cc:2258] T 33c90ae4cdf44bd8a7b6bfd7bb242279 P 902fcd0c4a5f4498b7d31bc1b28c62da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:17:06.973876 16173 ts_tablet_manager.cc:1905] T 33c90ae4cdf44bd8a7b6bfd7bb242279 P 902fcd0c4a5f4498b7d31bc1b28c62da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:17:06.979768 16175 tablet_replica.cc:330] T 33c90ae4cdf44bd8a7b6bfd7bb242279 P aa297e5b7e8242419fcffc6ced0b9874: stopping tablet replica
I20260924 16:17:06.980278 16175 raft_consensus.cc:2229] T 33c90ae4cdf44bd8a7b6bfd7bb242279 P aa297e5b7e8242419fcffc6ced0b9874 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:17:06.980904 16175 raft_consensus.cc:2258] T 33c90ae4cdf44bd8a7b6bfd7bb242279 P aa297e5b7e8242419fcffc6ced0b9874 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:17:06.981323 16174 ts_tablet_manager.cc:1918] T 33c90ae4cdf44bd8a7b6bfd7bb242279 P d32bf07da97342eb980c0ca29c7e8877: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:17:06.981621 16174 log.cc:1196] T 33c90ae4cdf44bd8a7b6bfd7bb242279 P d32bf07da97342eb980c0ca29c7e8877: Deleting WAL directory at /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/minicluster-data/ts-2/wal/wals/33c90ae4cdf44bd8a7b6bfd7bb242279
I20260924 16:17:06.982358 16174 ts_tablet_manager.cc:1939] T 33c90ae4cdf44bd8a7b6bfd7bb242279 P d32bf07da97342eb980c0ca29c7e8877: Deleting consensus metadata
I20260924 16:17:06.983572 16173 ts_tablet_manager.cc:1918] T 33c90ae4cdf44bd8a7b6bfd7bb242279 P 902fcd0c4a5f4498b7d31bc1b28c62da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:17:06.983891 16173 log.cc:1196] T 33c90ae4cdf44bd8a7b6bfd7bb242279 P 902fcd0c4a5f4498b7d31bc1b28c62da: Deleting WAL directory at /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/minicluster-data/ts-0/wal/wals/33c90ae4cdf44bd8a7b6bfd7bb242279
I20260924 16:17:06.984102 16175 ts_tablet_manager.cc:1905] T 33c90ae4cdf44bd8a7b6bfd7bb242279 P aa297e5b7e8242419fcffc6ced0b9874: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:17:06.984548 16173 ts_tablet_manager.cc:1939] T 33c90ae4cdf44bd8a7b6bfd7bb242279 P 902fcd0c4a5f4498b7d31bc1b28c62da: Deleting consensus metadata
I20260924 16:17:06.984500 15648 catalog_manager.cc:4841] TS d32bf07da97342eb980c0ca29c7e8877 (127.29.234.3:34041): tablet 33c90ae4cdf44bd8a7b6bfd7bb242279 (table test-table [id=fba85f7caeff448bb45cd2996094387d]) successfully deleted
I20260924 16:17:06.986624 15650 catalog_manager.cc:4841] TS 902fcd0c4a5f4498b7d31bc1b28c62da (127.29.234.1:39455): tablet 33c90ae4cdf44bd8a7b6bfd7bb242279 (table test-table [id=fba85f7caeff448bb45cd2996094387d]) successfully deleted
I20260924 16:17:06.989713 16175 ts_tablet_manager.cc:1918] T 33c90ae4cdf44bd8a7b6bfd7bb242279 P aa297e5b7e8242419fcffc6ced0b9874: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:17:06.989884 16175 log.cc:1196] T 33c90ae4cdf44bd8a7b6bfd7bb242279 P aa297e5b7e8242419fcffc6ced0b9874: Deleting WAL directory at /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790266323169505-30632-0/minicluster-data/ts-1/wal/wals/33c90ae4cdf44bd8a7b6bfd7bb242279
Sep 24 16:17:06 dist-test-slave-v5j9 krb5kdc[15619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.234.1@KRBTEST.COM
I20260924 16:17:06.990509 16175 ts_tablet_manager.cc:1939] T 33c90ae4cdf44bd8a7b6bfd7bb242279 P aa297e5b7e8242419fcffc6ced0b9874: Deleting consensus metadata
I20260924 16:17:06.992383 15648 catalog_manager.cc:4841] TS aa297e5b7e8242419fcffc6ced0b9874 (127.29.234.2:40813): tablet 33c90ae4cdf44bd8a7b6bfd7bb242279 (table test-table [id=fba85f7caeff448bb45cd2996094387d]) successfully deleted
I20260924 16:17:06.998921 16143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:06.956861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59266 (local address 127.29.234.1:39455)
0924 16:17:06.957059 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:06.957069 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:06.971812 (+ 14743us) server_negotiation.cc:407] Connection header received
0924 16:17:06.972202 (+ 390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:06.972222 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:06.972346 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:06.972440 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:06.974977 (+ 2537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:06.976570 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:06.981372 (+ 4802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:06.982066 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:06.982178 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:06.991733 (+ 9555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:06.991750 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:06.991770 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:17:06.991834 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:06.995728 (+ 3894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:17:06.996606 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:17:06.996617 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:17:06.996621 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:17:06.996804 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:17:06.997190 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:17:06.997196 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:17:06.997200 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:17:06.997739 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:06.998064 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:06.998273 (+ 209us) server_negotiation.cc:299] Negotiation successful
0924 16:17:06.998515 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 16:17:06.999656 15850 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:17:07.106540 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 15726
I20260924 16:17:07.132130 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 15861
Sep 24 16:17:07 dist-test-slave-v5j9 krb5kdc[15619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.234.2@KRBTEST.COM
I20260924 16:17:07.149515 16146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:07.133038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38748 (local address 127.29.234.2:40813)
0924 16:17:07.133215 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:07.133219 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:07.133258 (+ 39us) server_negotiation.cc:407] Connection header received
0924 16:17:07.133508 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:07.133513 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:07.133581 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:07.133728 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:07.135382 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:07.136135 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:07.137981 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:07.138368 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:07.138461 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:07.143631 (+ 5170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:07.143653 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:07.143673 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:17:07.143709 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:07.146621 (+ 2912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:17:07.147368 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:17:07.147377 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:17:07.147379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:17:07.147499 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:17:07.147902 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:17:07.147907 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:17:07.147909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:17:07.148281 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:07.148594 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:07.148888 (+ 294us) server_negotiation.cc:299] Negotiation successful
0924 16:17:07.149111 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 16:17:07.150211 15989 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:17:07.240667 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 15861
I20260924 16:17:07.266635 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 16000
Sep 24 16:17:07 dist-test-slave-v5j9 krb5kdc[15619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.234.3@KRBTEST.COM
I20260924 16:17:07.285758 16148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:07.267411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45458 (local address 127.29.234.3:34041)
0924 16:17:07.267594 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:07.267599 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:07.267711 (+ 112us) server_negotiation.cc:407] Connection header received
0924 16:17:07.267908 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:07.267912 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:07.267983 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:07.268074 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:07.270158 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:07.271248 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:07.273253 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:07.273804 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:07.273898 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:07.279353 (+ 5455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:07.279367 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:07.279387 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:17:07.279441 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:07.282688 (+ 3247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:17:07.283457 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:17:07.283466 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:17:07.283469 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:17:07.283649 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:17:07.284049 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:17:07.284053 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:17:07.284057 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:17:07.284465 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:07.284820 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:07.285119 (+ 299us) server_negotiation.cc:299] Negotiation successful
0924 16:17:07.285392 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 16:17:07.286432 16125 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:17:07.413159 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 16000
I20260924 16:17:07.433132 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 15632
I20260924 16:17:07.434409 15695 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:17:07.546177 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 15632
2026-09-24T16:17:07Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4368 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-24T16:17:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:17:07Z Disabled control of system clock
I20260924 16:17:07.626773 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266323169505-30632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266323169505-30632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266323169505-30632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:45367
--webserver_interface=127.29.234.62
--webserver_port=0
--builtin_ntp_servers=127.29.234.20:45531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.234.62:45367
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 16:17:07.965055 16184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:17:07.965389 16184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:17:07.965507 16184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:17:07.973230 16184 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:17:07.973378 16184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:17:07.973464 16184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:17:07.973526 16184 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:17:07.973570 16184 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:17:07.982714 16184 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:45531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266323169505-30632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266323169505-30632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.234.62:45367
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:45367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.234.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266323169505-30632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:17:07.984131 16184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:17:07.986328 16184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:17:07.998733 16192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:07.998823 16190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:07.998857 16189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:17:07.999475 16184 server_base.cc:1034] running on GCE node
I20260924 16:17:08.000223 16184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:17:08.001629 16184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:17:08.002888 16184 hybrid_clock.cc:648] HybridClock initialized: now 1790266628002848 us; error 29 us; skew 500 ppm
I20260924 16:17:08.003304 16184 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 16:17:08.005872 16184 webserver.cc:460] Webserver started at http://127.29.234.62:38479/ using document root <none> and password file <none>
I20260924 16:17:08.006608 16184 fs_manager.cc:359] Metadata directory not provided
I20260924 16:17:08.006732 16184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:17:08.007159 16184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:17:08.010248 16184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266323169505-30632-0/minicluster-data/master-0/data/instance:
uuid: "adccf10f80ca4dfe8d0fcc58653c71ad"
format_stamp: "Formatted at 2026-09-24 16:17:08 on dist-test-slave-v5j9"
server_key: "80ffa60bb47cd80214d103b873c923dd"
server_key_iv: "d36c35c32e5328ae61db0ab8bc1e3458"
server_key_version: "encryptionkey@0"
I20260924 16:17:08.011082 16184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266323169505-30632-0/minicluster-data/master-0/wal/instance:
uuid: "adccf10f80ca4dfe8d0fcc58653c71ad"
format_stamp: "Formatted at 2026-09-24 16:17:08 on dist-test-slave-v5j9"
server_key: "80ffa60bb47cd80214d103b873c923dd"
server_key_iv: "d36c35c32e5328ae61db0ab8bc1e3458"
server_key_version: "encryptionkey@0"
I20260924 16:17:08.016309 16184 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 16:17:08.019867 16199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:17:08.021142 16184 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 16:17:08.021337 16184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266323169505-30632-0/minicluster-data/master-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266323169505-30632-0/minicluster-data/master-0/wal
uuid: "adccf10f80ca4dfe8d0fcc58653c71ad"
format_stamp: "Formatted at 2026-09-24 16:17:08 on dist-test-slave-v5j9"
server_key: "80ffa60bb47cd80214d103b873c923dd"
server_key_iv: "d36c35c32e5328ae61db0ab8bc1e3458"
server_key_version: "encryptionkey@0"
I20260924 16:17:08.021622 16184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266323169505-30632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266323169505-30632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266323169505-30632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:17:08.036685 16184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:17:08.037693 16184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:17:08.038028 16184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:17:08.056577 16184 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.62:45367
I20260924 16:17:08.056598 16250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.62:45367 every 8 connection(s)
I20260924 16:17:08.058142 16184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
I20260924 16:17:08.060503 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 16184
I20260924 16:17:08.060691 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266323169505-30632-0/minicluster-data/master-0/wal/instance
I20260924 16:17:08.062994 16251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:17:08.066010 30632 external_mini_cluster.cc:1428] Setting key aad58c219e56f2283efb299259e309f7
I20260924 16:17:08.076397 16251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adccf10f80ca4dfe8d0fcc58653c71ad: Bootstrap starting.
I20260924 16:17:08.079453 16254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:08.068186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59096 (local address 127.29.234.62:45367)
0924 16:17:08.069199 (+ 1013us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:08.069214 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:08.069258 (+ 44us) server_negotiation.cc:407] Connection header received
0924 16:17:08.070869 (+ 1611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:08.070892 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:08.071365 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:08.071759 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:08.072327 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:08.073713 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:08.075042 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:08.075670 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:08.075861 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:08.077089 (+ 1228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:08.077115 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:08.077135 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:17:08.077738 (+ 603us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:08.077826 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:17:08.077851 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:08.077947 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:08.078207 (+ 260us) server_negotiation.cc:299] Negotiation successful
0924 16:17:08.078535 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":582,"thread_start_us":349,"threads_started":1}
I20260924 16:17:08.079929 16251 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P adccf10f80ca4dfe8d0fcc58653c71ad: Neither blocks nor log segments found. Creating new log.
I20260924 16:17:08.080971 16251 log.cc:824] T 00000000000000000000000000000000 P adccf10f80ca4dfe8d0fcc58653c71ad: Log is configured to *not* fsync() on all Append() calls
I20260924 16:17:08.083881 16251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adccf10f80ca4dfe8d0fcc58653c71ad: No bootstrap required, opened a new log
I20260924 16:17:08.091308 16251 raft_consensus.cc:348] T 00000000000000000000000000000000 P adccf10f80ca4dfe8d0fcc58653c71ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adccf10f80ca4dfe8d0fcc58653c71ad" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 45367 } }
I20260924 16:17:08.091739 16251 raft_consensus.cc:374] T 00000000000000000000000000000000 P adccf10f80ca4dfe8d0fcc58653c71ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:17:08.091863 16251 raft_consensus.cc:729] T 00000000000000000000000000000000 P adccf10f80ca4dfe8d0fcc58653c71ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adccf10f80ca4dfe8d0fcc58653c71ad, State: Initialized, Role: FOLLOWER
I20260924 16:17:08.092530 16251 consensus_queue.cc:260] T 00000000000000000000000000000000 P adccf10f80ca4dfe8d0fcc58653c71ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adccf10f80ca4dfe8d0fcc58653c71ad" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 45367 } }
I20260924 16:17:08.092785 16251 raft_consensus.cc:388] T 00000000000000000000000000000000 P adccf10f80ca4dfe8d0fcc58653c71ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:17:08.092932 16251 raft_consensus.cc:482] T 00000000000000000000000000000000 P adccf10f80ca4dfe8d0fcc58653c71ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:17:08.093118 16251 raft_consensus.cc:3037] T 00000000000000000000000000000000 P adccf10f80ca4dfe8d0fcc58653c71ad [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:17:08.095682 16251 raft_consensus.cc:504] T 00000000000000000000000000000000 P adccf10f80ca4dfe8d0fcc58653c71ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adccf10f80ca4dfe8d0fcc58653c71ad" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 45367 } }
I20260924 16:17:08.096194 16251 leader_election.cc:302] T 00000000000000000000000000000000 P adccf10f80ca4dfe8d0fcc58653c71ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: adccf10f80ca4dfe8d0fcc58653c71ad; no voters:
I20260924 16:17:08.096832 16251 leader_election.cc:288] T 00000000000000000000000000000000 P adccf10f80ca4dfe8d0fcc58653c71ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:17:08.096989 16256 raft_consensus.cc:2781] T 00000000000000000000000000000000 P adccf10f80ca4dfe8d0fcc58653c71ad [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:17:08.098235 16256 raft_consensus.cc:686] T 00000000000000000000000000000000 P adccf10f80ca4dfe8d0fcc58653c71ad [term 1 LEADER]: Becoming Leader. State: Replica: adccf10f80ca4dfe8d0fcc58653c71ad, State: Running, Role: LEADER
I20260924 16:17:08.098894 16256 consensus_queue.cc:237] T 00000000000000000000000000000000 P adccf10f80ca4dfe8d0fcc58653c71ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adccf10f80ca4dfe8d0fcc58653c71ad" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 45367 } }
I20260924 16:17:08.099251 16251 sys_catalog.cc:564] T 00000000000000000000000000000000 P adccf10f80ca4dfe8d0fcc58653c71ad [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:17:08.101660 16258 sys_catalog.cc:455] T 00000000000000000000000000000000 P adccf10f80ca4dfe8d0fcc58653c71ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader adccf10f80ca4dfe8d0fcc58653c71ad. Latest consensus state: current_term: 1 leader_uuid: "adccf10f80ca4dfe8d0fcc58653c71ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adccf10f80ca4dfe8d0fcc58653c71ad" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 45367 } } }
I20260924 16:17:08.102144 16258 sys_catalog.cc:458] T 00000000000000000000000000000000 P adccf10f80ca4dfe8d0fcc58653c71ad [sys.catalog]: This master's current role is: LEADER
I20260924 16:17:08.102731 16257 sys_catalog.cc:455] T 00000000000000000000000000000000 P adccf10f80ca4dfe8d0fcc58653c71ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "adccf10f80ca4dfe8d0fcc58653c71ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adccf10f80ca4dfe8d0fcc58653c71ad" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 45367 } } }
I20260924 16:17:08.103098 16257 sys_catalog.cc:458] T 00000000000000000000000000000000 P adccf10f80ca4dfe8d0fcc58653c71ad [sys.catalog]: This master's current role is: LEADER
I20260924 16:17:08.103137 16265 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:17:08.110091 16265 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:17:08.118688 16265 catalog_manager.cc:1347] Generated new cluster ID: 620ab375ad8044e3a1def782b2366882
I20260924 16:17:08.118819 16265 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:17:08.132467 16265 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:17:08.134246 16265 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:17:08.146018 16265 catalog_manager.cc:5868] T 00000000000000000000000000000000 P adccf10f80ca4dfe8d0fcc58653c71ad: Generated new TSK 0
I20260924 16:17:08.147161 16265 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:17:08.162564 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266323169505-30632-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266323169505-30632-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266323169505-30632-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.1:0
--local_ip_for_outbound_sockets=127.29.234.1
--webserver_interface=127.29.234.1
--webserver_port=0
--tserver_master_addrs=127.29.234.62:45367
--builtin_ntp_servers=127.29.234.20:45531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 16:17:08.498497 16276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:17:08.498803 16276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:17:08.498893 16276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:17:08.506511 16276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:17:08.506618 16276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:17:08.506724 16276 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.1
I20260924 16:17:08.516075 16276 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:45531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266323169505-30632-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266323169505-30632-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.234.1
--webserver_port=0
--tserver_master_addrs=127.29.234.62:45367
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.1
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266323169505-30632-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:17:08.517328 16276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:17:08.519090 16276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:17:08.532241 16284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:08.532321 16281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:08.532290 16282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:17:08.533329 16276 server_base.cc:1034] running on GCE node
I20260924 16:17:08.534022 16276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:17:08.534971 16276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:17:08.536208 16276 hybrid_clock.cc:648] HybridClock initialized: now 1790266628536159 us; error 41 us; skew 500 ppm
I20260924 16:17:08.536545 16276 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 16:17:08.538950 16276 webserver.cc:460] Webserver started at http://127.29.234.1:46459/ using document root <none> and password file <none>
I20260924 16:17:08.539726 16276 fs_manager.cc:359] Metadata directory not provided
I20260924 16:17:08.539858 16276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:17:08.540177 16276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:17:08.543150 16276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266323169505-30632-0/minicluster-data/ts-0/data/instance:
uuid: "c2f691b72da449edb2c24eff0dac1cdc"
format_stamp: "Formatted at 2026-09-24 16:17:08 on dist-test-slave-v5j9"
server_key: "c95ec020054e7992ff0770470080ee7d"
server_key_iv: "9424dce299fb1471916f6a670583112e"
server_key_version: "encryptionkey@0"
I20260924 16:17:08.543987 16276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266323169505-30632-0/minicluster-data/ts-0/wal/instance:
uuid: "c2f691b72da449edb2c24eff0dac1cdc"
format_stamp: "Formatted at 2026-09-24 16:17:08 on dist-test-slave-v5j9"
server_key: "c95ec020054e7992ff0770470080ee7d"
server_key_iv: "9424dce299fb1471916f6a670583112e"
server_key_version: "encryptionkey@0"
I20260924 16:17:08.549117 16276 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 16:17:08.552500 16290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:17:08.553915 16276 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:17:08.554087 16276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266323169505-30632-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266323169505-30632-0/minicluster-data/ts-0/wal
uuid: "c2f691b72da449edb2c24eff0dac1cdc"
format_stamp: "Formatted at 2026-09-24 16:17:08 on dist-test-slave-v5j9"
server_key: "c95ec020054e7992ff0770470080ee7d"
server_key_iv: "9424dce299fb1471916f6a670583112e"
server_key_version: "encryptionkey@0"
I20260924 16:17:08.554319 16276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266323169505-30632-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266323169505-30632-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266323169505-30632-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:17:08.569327 16276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:17:08.570241 16276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:17:08.570566 16276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:17:08.571574 16276 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:17:08.573017 16276 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:17:08.573138 16276 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:17:08.573276 16276 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:17:08.573386 16276 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:17:08.606912 16276 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.1:42223
I20260924 16:17:08.607028 16404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.1:42223 every 8 connection(s)
I20260924 16:17:08.608369 16276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
I20260924 16:17:08.617328 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 16276
I20260924 16:17:08.617571 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266323169505-30632-0/minicluster-data/ts-0/wal/instance
I20260924 16:17:08.618036 30632 external_mini_cluster.cc:1428] Setting key e374ea0a2f6453b8d52d5a6d2aaac457
I20260924 16:17:08.621031 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266323169505-30632-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266323169505-30632-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266323169505-30632-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.2:0
--local_ip_for_outbound_sockets=127.29.234.2
--webserver_interface=127.29.234.2
--webserver_port=0
--tserver_master_addrs=127.29.234.62:45367
--builtin_ntp_servers=127.29.234.20:45531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 16:17:08.621728 16407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:08.611218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.1:46955 (local address 127.29.234.62:45367)
0924 16:17:08.611657 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:08.611662 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:08.612867 (+ 1205us) server_negotiation.cc:407] Connection header received
0924 16:17:08.614793 (+ 1926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:08.614799 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:08.614896 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:08.615032 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:08.616161 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:08.616891 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:08.618669 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:08.619190 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:08.619311 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:08.620748 (+ 1437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:08.620757 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:08.620764 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:17:08.620787 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:08.620849 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:17:08.620867 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:08.620976 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:08.621173 (+ 197us) server_negotiation.cc:299] Negotiation successful
0924 16:17:08.621292 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":225,"threads_started":1}
I20260924 16:17:08.622614 16408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:08.611826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:45367 (local address 127.29.234.1:46955)
0924 16:17:08.612717 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0924 16:17:08.612754 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 16:17:08.614492 (+ 1738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:17:08.615287 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:17:08.615296 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:17:08.615389 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:17:08.615986 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:08.616004 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:08.617068 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:08.617078 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:08.618522 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:08.618535 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:08.619409 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:08.619415 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:08.619687 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:08.620428 (+ 741us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:17:08.620431 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:17:08.620469 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:17:08.620546 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:17:08.620967 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:17:08.620975 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:17:08.620981 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 16:17:08.621149 (+ 168us) client_negotiation.cc:240] Negotiation successful
0924 16:17:08.621553 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":265,"threads_started":1}
I20260924 16:17:08.624133 16405 heartbeater.cc:347] Connected to a master server at 127.29.234.62:45367
I20260924 16:17:08.624454 16405 heartbeater.cc:464] Registering TS with master...
I20260924 16:17:08.625468 16405 heartbeater.cc:511] Master 127.29.234.62:45367 requested a full tablet report, sending...
I20260924 16:17:08.627631 16216 ts_manager.cc:194] Registered new tserver with Master: c2f691b72da449edb2c24eff0dac1cdc (127.29.234.1:42223)
I20260924 16:17:08.629109 16216 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.234.1:46955
W20260924 16:17:08.956144 16409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:17:08.956414 16409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:17:08.956584 16409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:17:08.963824 16409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:17:08.963917 16409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:17:08.964109 16409 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.2
I20260924 16:17:08.973093 16409 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:45531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266323169505-30632-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266323169505-30632-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.234.2
--webserver_port=0
--tserver_master_addrs=127.29.234.62:45367
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.2
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266323169505-30632-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:17:08.974465 16409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:17:08.976382 16409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:17:08.990025 16414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:08.990676 16415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:08.990711 16417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:17:08.990736 16409 server_base.cc:1034] running on GCE node
I20260924 16:17:08.991494 16409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:17:08.992388 16409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:17:08.993623 16409 hybrid_clock.cc:648] HybridClock initialized: now 1790266628993568 us; error 45 us; skew 500 ppm
I20260924 16:17:08.993968 16409 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 16:17:08.996507 16409 webserver.cc:460] Webserver started at http://127.29.234.2:45877/ using document root <none> and password file <none>
I20260924 16:17:08.997258 16409 fs_manager.cc:359] Metadata directory not provided
I20260924 16:17:08.997417 16409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:17:08.997748 16409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:17:09.000918 16409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266323169505-30632-0/minicluster-data/ts-1/data/instance:
uuid: "1e2c01377a4f4d10877d8d4ab8297e2b"
format_stamp: "Formatted at 2026-09-24 16:17:08 on dist-test-slave-v5j9"
server_key: "4f57ef23149f07e365600fa83d1ed419"
server_key_iv: "510f442897a1d7760d7272777c95307c"
server_key_version: "encryptionkey@0"
I20260924 16:17:09.001827 16409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266323169505-30632-0/minicluster-data/ts-1/wal/instance:
uuid: "1e2c01377a4f4d10877d8d4ab8297e2b"
format_stamp: "Formatted at 2026-09-24 16:17:08 on dist-test-slave-v5j9"
server_key: "4f57ef23149f07e365600fa83d1ed419"
server_key_iv: "510f442897a1d7760d7272777c95307c"
server_key_version: "encryptionkey@0"
I20260924 16:17:09.007575 16409 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260924 16:17:09.011729 16423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:17:09.013032 16409 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 16:17:09.013221 16409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266323169505-30632-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266323169505-30632-0/minicluster-data/ts-1/wal
uuid: "1e2c01377a4f4d10877d8d4ab8297e2b"
format_stamp: "Formatted at 2026-09-24 16:17:08 on dist-test-slave-v5j9"
server_key: "4f57ef23149f07e365600fa83d1ed419"
server_key_iv: "510f442897a1d7760d7272777c95307c"
server_key_version: "encryptionkey@0"
I20260924 16:17:09.013499 16409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266323169505-30632-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266323169505-30632-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266323169505-30632-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:17:09.034667 16409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:17:09.035501 16409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:17:09.035815 16409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:17:09.036816 16409 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:17:09.038401 16409 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:17:09.038524 16409 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:17:09.038662 16409 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:17:09.038753 16409 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:17:09.072893 16409 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.2:44079
I20260924 16:17:09.072959 16537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.2:44079 every 8 connection(s)
I20260924 16:17:09.074280 16409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
I20260924 16:17:09.077111 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 16409
I20260924 16:17:09.077296 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266323169505-30632-0/minicluster-data/ts-1/wal/instance
I20260924 16:17:09.077900 30632 external_mini_cluster.cc:1428] Setting key 657dc5093eb52dc94f4a25821734fe33
I20260924 16:17:09.081081 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266323169505-30632-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266323169505-30632-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266323169505-30632-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.3:0
--local_ip_for_outbound_sockets=127.29.234.3
--webserver_interface=127.29.234.3
--webserver_port=0
--tserver_master_addrs=127.29.234.62:45367
--builtin_ntp_servers=127.29.234.20:45531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 16:17:09.088728 16407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:09.077254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.2:44611 (local address 127.29.234.62:45367)
0924 16:17:09.077462 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:09.077467 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:09.079364 (+ 1897us) server_negotiation.cc:407] Connection header received
0924 16:17:09.081515 (+ 2151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:09.081523 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:09.081618 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:09.081739 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:09.082827 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:09.083957 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:09.085582 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:09.086350 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:09.086487 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:09.087798 (+ 1311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:09.087806 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:09.087814 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:17:09.087839 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:09.087882 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:17:09.087900 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:09.087990 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:09.088210 (+ 220us) server_negotiation.cc:299] Negotiation successful
0924 16:17:09.088333 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 16:17:09.089854 16540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:09.078012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:45367 (local address 127.29.234.2:44611)
0924 16:17:09.079189 (+ 1177us) negotiation.cc:107] Waiting for socket to connect
0924 16:17:09.079252 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0924 16:17:09.081127 (+ 1875us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:17:09.081938 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:17:09.081947 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:17:09.082043 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:17:09.082647 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:09.082665 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:09.084125 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:09.084129 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:09.085445 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:09.085457 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:09.086552 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:09.086556 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:09.086762 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:09.087484 (+ 722us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:17:09.087486 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:17:09.087525 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:17:09.087605 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:17:09.087992 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:17:09.087999 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:17:09.088022 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 16:17:09.088173 (+ 151us) client_negotiation.cc:240] Negotiation successful
0924 16:17:09.088476 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":358,"threads_started":1}
I20260924 16:17:09.091352 16538 heartbeater.cc:347] Connected to a master server at 127.29.234.62:45367
I20260924 16:17:09.091678 16538 heartbeater.cc:464] Registering TS with master...
I20260924 16:17:09.092448 16538 heartbeater.cc:511] Master 127.29.234.62:45367 requested a full tablet report, sending...
I20260924 16:17:09.093885 16216 ts_manager.cc:194] Registered new tserver with Master: 1e2c01377a4f4d10877d8d4ab8297e2b (127.29.234.2:44079)
I20260924 16:17:09.094705 16216 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.234.2:44611
W20260924 16:17:09.409024 16541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:17:09.409390 16541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:17:09.409513 16541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:17:09.417136 16541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:17:09.417253 16541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:17:09.417425 16541 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.3
I20260924 16:17:09.428906 16541 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:45531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266323169505-30632-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266323169505-30632-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.234.3
--webserver_port=0
--tserver_master_addrs=127.29.234.62:45367
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.3
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266323169505-30632-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:17:09.430317 16541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:17:09.432083 16541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:17:09.445302 16546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:09.445323 16549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:09.445377 16547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:17:09.446216 16541 server_base.cc:1034] running on GCE node
I20260924 16:17:09.446861 16541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:17:09.447863 16541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:17:09.449100 16541 hybrid_clock.cc:648] HybridClock initialized: now 1790266629449057 us; error 55 us; skew 500 ppm
I20260924 16:17:09.449503 16541 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 16:17:09.452131 16541 webserver.cc:460] Webserver started at http://127.29.234.3:38149/ using document root <none> and password file <none>
I20260924 16:17:09.452886 16541 fs_manager.cc:359] Metadata directory not provided
I20260924 16:17:09.453015 16541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:17:09.453402 16541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:17:09.456358 16541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266323169505-30632-0/minicluster-data/ts-2/data/instance:
uuid: "c61dd0eaba9542068c71c4a81990d779"
format_stamp: "Formatted at 2026-09-24 16:17:09 on dist-test-slave-v5j9"
server_key: "d6c3c84b10200a10ade6db637a262f95"
server_key_iv: "2e59556c25dacf21b53ba2e9c5c29191"
server_key_version: "encryptionkey@0"
I20260924 16:17:09.457166 16541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266323169505-30632-0/minicluster-data/ts-2/wal/instance:
uuid: "c61dd0eaba9542068c71c4a81990d779"
format_stamp: "Formatted at 2026-09-24 16:17:09 on dist-test-slave-v5j9"
server_key: "d6c3c84b10200a10ade6db637a262f95"
server_key_iv: "2e59556c25dacf21b53ba2e9c5c29191"
server_key_version: "encryptionkey@0"
I20260924 16:17:09.462482 16541 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 16:17:09.466050 16555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:17:09.467423 16541 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:17:09.467608 16541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266323169505-30632-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266323169505-30632-0/minicluster-data/ts-2/wal
uuid: "c61dd0eaba9542068c71c4a81990d779"
format_stamp: "Formatted at 2026-09-24 16:17:09 on dist-test-slave-v5j9"
server_key: "d6c3c84b10200a10ade6db637a262f95"
server_key_iv: "2e59556c25dacf21b53ba2e9c5c29191"
server_key_version: "encryptionkey@0"
I20260924 16:17:09.467836 16541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266323169505-30632-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266323169505-30632-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266323169505-30632-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:17:09.478830 16541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:17:09.479641 16541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:17:09.479951 16541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:17:09.480922 16541 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:17:09.482429 16541 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:17:09.482550 16541 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:17:09.482695 16541 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:17:09.482784 16541 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:17:09.515635 16541 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.3:38663
I20260924 16:17:09.515723 16670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.3:38663 every 8 connection(s)
I20260924 16:17:09.517076 16541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
I20260924 16:17:09.527073 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 16541
I20260924 16:17:09.527267 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266323169505-30632-0/minicluster-data/ts-2/wal/instance
I20260924 16:17:09.527771 30632 external_mini_cluster.cc:1428] Setting key fce9e2613a0a203a87ccf149500c05bf
I20260924 16:17:09.529585 16407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:09.519703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.3:48197 (local address 127.29.234.62:45367)
0924 16:17:09.519909 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:09.519914 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:09.521299 (+ 1385us) server_negotiation.cc:407] Connection header received
0924 16:17:09.523084 (+ 1785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:09.523090 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:09.523180 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:09.523288 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:09.524328 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:09.525081 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:09.526721 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:09.527179 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:09.527319 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:09.528722 (+ 1403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:09.528727 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:09.528730 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:17:09.528747 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:09.528782 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:17:09.528796 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:09.528889 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:09.529110 (+ 221us) server_negotiation.cc:299] Negotiation successful
0924 16:17:09.529206 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260924 16:17:09.530537 16673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:09.520264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:45367 (local address 127.29.234.3:48197)
0924 16:17:09.521144 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0924 16:17:09.521179 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 16:17:09.522815 (+ 1636us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:17:09.523529 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:17:09.523538 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:17:09.523627 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:17:09.524166 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:09.524182 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:09.525229 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:09.525239 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:09.526586 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:09.526598 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:09.527420 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:09.527428 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:09.527653 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:09.528415 (+ 762us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:17:09.528418 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:17:09.528460 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:17:09.528533 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:17:09.528889 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:17:09.528901 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:17:09.528912 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 16:17:09.529102 (+ 190us) client_negotiation.cc:240] Negotiation successful
0924 16:17:09.529515 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":280,"threads_started":1}
I20260924 16:17:09.531806 16671 heartbeater.cc:347] Connected to a master server at 127.29.234.62:45367
I20260924 16:17:09.532094 16671 heartbeater.cc:464] Registering TS with master...
I20260924 16:17:09.532799 16671 heartbeater.cc:511] Master 127.29.234.62:45367 requested a full tablet report, sending...
I20260924 16:17:09.534169 16216 ts_manager.cc:194] Registered new tserver with Master: c61dd0eaba9542068c71c4a81990d779 (127.29.234.3:38663)
I20260924 16:17:09.534901 16216 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.234.3:48197
I20260924 16:17:09.543115 30632 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:17:09.552449 16407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:09.547073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59100 (local address 127.29.234.62:45367)
0924 16:17:09.547358 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:09.547365 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:09.547613 (+ 248us) server_negotiation.cc:407] Connection header received
0924 16:17:09.547743 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:09.547747 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:09.547819 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:09.547896 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:09.548360 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:09.549047 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:09.550428 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:09.550803 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:09.550885 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:09.551814 (+ 929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:09.551818 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:09.551821 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:17:09.551836 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:09.551877 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:17:09.551888 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:09.551948 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:09.552059 (+ 111us) server_negotiation.cc:299] Negotiation successful
0924 16:17:09.552124 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 16:17:09.555958 16216 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59100:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:17:09.563102 16216 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:17:09.595531 16685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:09.575161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49156 (local address 127.29.234.2:44079)
0924 16:17:09.586479 (+ 11318us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:09.586489 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:09.586526 (+ 37us) server_negotiation.cc:407] Connection header received
0924 16:17:09.586634 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:09.586642 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:09.586926 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:09.587170 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:17:09.588050 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:09.589657 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:09.592499 (+ 2842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:09.594023 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:09.594146 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:09.594412 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:09.595019 (+ 607us) server_negotiation.cc:299] Negotiation successful
0924 16:17:09.595195 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11137,"thread_start_us":11004,"threads_started":1}
I20260924 16:17:09.595512 16684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:09.574868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.2:44079 (local address 127.0.0.1:49156)
0924 16:17:09.585628 (+ 10760us) negotiation.cc:107] Waiting for socket to connect
0924 16:17:09.585692 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0924 16:17:09.585930 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:17:09.587283 (+ 1353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:17:09.587292 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:17:09.587331 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:17:09.587869 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:09.587886 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:09.589819 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:09.589824 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:09.592264 (+ 2440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:09.592282 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:09.594535 (+ 2253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:09.594541 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:09.594762 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:09.594827 (+ 65us) client_negotiation.cc:769] Sending connection context
0924 16:17:09.594975 (+ 148us) client_negotiation.cc:240] Negotiation successful
0924 16:17:09.595162 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10349,"thread_start_us":12863,"threads_started":1}
I20260924 16:17:09.596045 16683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:09.574913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.3:38663 (local address 127.0.0.1:36272)
0924 16:17:09.588058 (+ 13145us) negotiation.cc:107] Waiting for socket to connect
0924 16:17:09.588078 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 16:17:09.588270 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:17:09.589120 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:17:09.589126 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:17:09.589151 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:17:09.589645 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:09.589657 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:09.591681 (+ 2024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:09.591687 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:09.593609 (+ 1922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:09.593622 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:09.595304 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:09.595312 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:09.595513 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:09.595550 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 16:17:09.595624 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 16:17:09.595748 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13005,"thread_start_us":10270,"threads_started":1}
I20260924 16:17:09.596858 16686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:09.575578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36272 (local address 127.29.234.3:38663)
0924 16:17:09.586859 (+ 11281us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:09.586876 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:09.588171 (+ 1295us) server_negotiation.cc:407] Connection header received
0924 16:17:09.588443 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:09.588453 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:09.588678 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:09.588872 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:17:09.589842 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:09.591512 (+ 1670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:09.593809 (+ 2297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:09.594723 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:09.594880 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:09.595175 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:09.596309 (+ 1134us) server_negotiation.cc:299] Negotiation successful
0924 16:17:09.596491 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11114,"thread_start_us":10965,"threads_started":1}
I20260924 16:17:09.597096 16689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:09.588226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:42223 (local address 127.0.0.1:51850)
0924 16:17:09.589538 (+ 1312us) negotiation.cc:107] Waiting for socket to connect
0924 16:17:09.589555 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:17:09.589707 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:17:09.590385 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:17:09.590394 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:17:09.590421 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:17:09.590859 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:09.590874 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:09.592567 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:09.592572 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:09.594639 (+ 2067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:09.594652 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:09.596536 (+ 1884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:09.596540 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:09.596656 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:09.596678 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 16:17:09.596735 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 16:17:09.596822 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1213,"thread_start_us":1019,"threads_started":1}
I20260924 16:17:09.597283 16687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:09.588464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51850 (local address 127.29.234.1:42223)
0924 16:17:09.589001 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:09.589010 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:09.589670 (+ 660us) server_negotiation.cc:407] Connection header received
0924 16:17:09.589966 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:09.589974 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:09.590190 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:09.590374 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:17:09.591031 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:09.592415 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:09.594852 (+ 2437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:09.596355 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:09.596482 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:09.596717 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:09.596786 (+ 69us) server_negotiation.cc:299] Negotiation successful
0924 16:17:09.596960 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":273,"threads_started":1}
I20260924 16:17:09.601282 16604 tablet_service.cc:1467] Processing CreateTablet for tablet 25f519a0857e4f17993ad70a4caae2ef (DEFAULT_TABLE table=test-table [id=190306af6eba4741bc7bd61f6487f20b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:17:09.601527 16339 tablet_service.cc:1467] Processing CreateTablet for tablet 25f519a0857e4f17993ad70a4caae2ef (DEFAULT_TABLE table=test-table [id=190306af6eba4741bc7bd61f6487f20b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:17:09.601704 16472 tablet_service.cc:1467] Processing CreateTablet for tablet 25f519a0857e4f17993ad70a4caae2ef (DEFAULT_TABLE table=test-table [id=190306af6eba4741bc7bd61f6487f20b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:17:09.602900 16604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25f519a0857e4f17993ad70a4caae2ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:17:09.602896 16472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25f519a0857e4f17993ad70a4caae2ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:17:09.602881 16339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25f519a0857e4f17993ad70a4caae2ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:17:09.622220 16690 tablet_bootstrap.cc:492] T 25f519a0857e4f17993ad70a4caae2ef P c61dd0eaba9542068c71c4a81990d779: Bootstrap starting.
I20260924 16:17:09.623507 16691 tablet_bootstrap.cc:492] T 25f519a0857e4f17993ad70a4caae2ef P c2f691b72da449edb2c24eff0dac1cdc: Bootstrap starting.
I20260924 16:17:09.626253 16690 tablet_bootstrap.cc:654] T 25f519a0857e4f17993ad70a4caae2ef P c61dd0eaba9542068c71c4a81990d779: Neither blocks nor log segments found. Creating new log.
I20260924 16:17:09.627509 16690 log.cc:824] T 25f519a0857e4f17993ad70a4caae2ef P c61dd0eaba9542068c71c4a81990d779: Log is configured to *not* fsync() on all Append() calls
I20260924 16:17:09.627807 16691 tablet_bootstrap.cc:654] T 25f519a0857e4f17993ad70a4caae2ef P c2f691b72da449edb2c24eff0dac1cdc: Neither blocks nor log segments found. Creating new log.
I20260924 16:17:09.628676 16692 tablet_bootstrap.cc:492] T 25f519a0857e4f17993ad70a4caae2ef P 1e2c01377a4f4d10877d8d4ab8297e2b: Bootstrap starting.
I20260924 16:17:09.629006 16691 log.cc:824] T 25f519a0857e4f17993ad70a4caae2ef P c2f691b72da449edb2c24eff0dac1cdc: Log is configured to *not* fsync() on all Append() calls
I20260924 16:17:09.631438 16690 tablet_bootstrap.cc:492] T 25f519a0857e4f17993ad70a4caae2ef P c61dd0eaba9542068c71c4a81990d779: No bootstrap required, opened a new log
I20260924 16:17:09.631932 16405 heartbeater.cc:503] Master 127.29.234.62:45367 was elected leader, sending a full tablet report...
I20260924 16:17:09.632850 16692 tablet_bootstrap.cc:654] T 25f519a0857e4f17993ad70a4caae2ef P 1e2c01377a4f4d10877d8d4ab8297e2b: Neither blocks nor log segments found. Creating new log.
I20260924 16:17:09.633014 16690 ts_tablet_manager.cc:1397] T 25f519a0857e4f17993ad70a4caae2ef P c61dd0eaba9542068c71c4a81990d779: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260924 16:17:09.634410 16692 log.cc:824] T 25f519a0857e4f17993ad70a4caae2ef P 1e2c01377a4f4d10877d8d4ab8297e2b: Log is configured to *not* fsync() on all Append() calls
I20260924 16:17:09.638432 16691 tablet_bootstrap.cc:492] T 25f519a0857e4f17993ad70a4caae2ef P c2f691b72da449edb2c24eff0dac1cdc: No bootstrap required, opened a new log
I20260924 16:17:09.638751 16691 ts_tablet_manager.cc:1397] T 25f519a0857e4f17993ad70a4caae2ef P c2f691b72da449edb2c24eff0dac1cdc: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.006s
I20260924 16:17:09.639432 16692 tablet_bootstrap.cc:492] T 25f519a0857e4f17993ad70a4caae2ef P 1e2c01377a4f4d10877d8d4ab8297e2b: No bootstrap required, opened a new log
I20260924 16:17:09.639730 16692 ts_tablet_manager.cc:1397] T 25f519a0857e4f17993ad70a4caae2ef P 1e2c01377a4f4d10877d8d4ab8297e2b: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20260924 16:17:09.643301 16690 raft_consensus.cc:348] T 25f519a0857e4f17993ad70a4caae2ef P c61dd0eaba9542068c71c4a81990d779 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2c01377a4f4d10877d8d4ab8297e2b" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 44079 } } peers { permanent_uuid: "c61dd0eaba9542068c71c4a81990d779" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 38663 } } peers { permanent_uuid: "c2f691b72da449edb2c24eff0dac1cdc" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 42223 } }
I20260924 16:17:09.643733 16690 raft_consensus.cc:374] T 25f519a0857e4f17993ad70a4caae2ef P c61dd0eaba9542068c71c4a81990d779 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:17:09.643868 16690 raft_consensus.cc:729] T 25f519a0857e4f17993ad70a4caae2ef P c61dd0eaba9542068c71c4a81990d779 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c61dd0eaba9542068c71c4a81990d779, State: Initialized, Role: FOLLOWER
I20260924 16:17:09.644523 16690 consensus_queue.cc:260] T 25f519a0857e4f17993ad70a4caae2ef P c61dd0eaba9542068c71c4a81990d779 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2c01377a4f4d10877d8d4ab8297e2b" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 44079 } } peers { permanent_uuid: "c61dd0eaba9542068c71c4a81990d779" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 38663 } } peers { permanent_uuid: "c2f691b72da449edb2c24eff0dac1cdc" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 42223 } }
I20260924 16:17:09.645795 16671 heartbeater.cc:503] Master 127.29.234.62:45367 was elected leader, sending a full tablet report...
I20260924 16:17:09.646461 16690 ts_tablet_manager.cc:1428] T 25f519a0857e4f17993ad70a4caae2ef P c61dd0eaba9542068c71c4a81990d779: Time spent starting tablet: real 0.013s user 0.007s sys 0.005s
I20260924 16:17:09.647791 16692 raft_consensus.cc:348] T 25f519a0857e4f17993ad70a4caae2ef P 1e2c01377a4f4d10877d8d4ab8297e2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2c01377a4f4d10877d8d4ab8297e2b" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 44079 } } peers { permanent_uuid: "c61dd0eaba9542068c71c4a81990d779" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 38663 } } peers { permanent_uuid: "c2f691b72da449edb2c24eff0dac1cdc" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 42223 } }
I20260924 16:17:09.648114 16692 raft_consensus.cc:374] T 25f519a0857e4f17993ad70a4caae2ef P 1e2c01377a4f4d10877d8d4ab8297e2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:17:09.648229 16692 raft_consensus.cc:729] T 25f519a0857e4f17993ad70a4caae2ef P 1e2c01377a4f4d10877d8d4ab8297e2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e2c01377a4f4d10877d8d4ab8297e2b, State: Initialized, Role: FOLLOWER
I20260924 16:17:09.648864 16692 consensus_queue.cc:260] T 25f519a0857e4f17993ad70a4caae2ef P 1e2c01377a4f4d10877d8d4ab8297e2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2c01377a4f4d10877d8d4ab8297e2b" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 44079 } } peers { permanent_uuid: "c61dd0eaba9542068c71c4a81990d779" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 38663 } } peers { permanent_uuid: "c2f691b72da449edb2c24eff0dac1cdc" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 42223 } }
I20260924 16:17:09.649005 16691 raft_consensus.cc:348] T 25f519a0857e4f17993ad70a4caae2ef P c2f691b72da449edb2c24eff0dac1cdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2c01377a4f4d10877d8d4ab8297e2b" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 44079 } } peers { permanent_uuid: "c61dd0eaba9542068c71c4a81990d779" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 38663 } } peers { permanent_uuid: "c2f691b72da449edb2c24eff0dac1cdc" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 42223 } }
I20260924 16:17:09.649477 16691 raft_consensus.cc:374] T 25f519a0857e4f17993ad70a4caae2ef P c2f691b72da449edb2c24eff0dac1cdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:17:09.649588 16691 raft_consensus.cc:729] T 25f519a0857e4f17993ad70a4caae2ef P c2f691b72da449edb2c24eff0dac1cdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2f691b72da449edb2c24eff0dac1cdc, State: Initialized, Role: FOLLOWER
I20260924 16:17:09.650123 16691 consensus_queue.cc:260] T 25f519a0857e4f17993ad70a4caae2ef P c2f691b72da449edb2c24eff0dac1cdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2c01377a4f4d10877d8d4ab8297e2b" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 44079 } } peers { permanent_uuid: "c61dd0eaba9542068c71c4a81990d779" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 38663 } } peers { permanent_uuid: "c2f691b72da449edb2c24eff0dac1cdc" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 42223 } }
I20260924 16:17:09.651762 16691 ts_tablet_manager.cc:1428] T 25f519a0857e4f17993ad70a4caae2ef P c2f691b72da449edb2c24eff0dac1cdc: Time spent starting tablet: real 0.013s user 0.009s sys 0.002s
I20260924 16:17:09.652006 16538 heartbeater.cc:503] Master 127.29.234.62:45367 was elected leader, sending a full tablet report...
I20260924 16:17:09.652627 16692 ts_tablet_manager.cc:1428] T 25f519a0857e4f17993ad70a4caae2ef P 1e2c01377a4f4d10877d8d4ab8297e2b: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260924 16:17:09.673744 16699 raft_consensus.cc:482] T 25f519a0857e4f17993ad70a4caae2ef P c2f691b72da449edb2c24eff0dac1cdc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:17:09.674180 16699 raft_consensus.cc:504] T 25f519a0857e4f17993ad70a4caae2ef P c2f691b72da449edb2c24eff0dac1cdc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2c01377a4f4d10877d8d4ab8297e2b" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 44079 } } peers { permanent_uuid: "c61dd0eaba9542068c71c4a81990d779" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 38663 } } peers { permanent_uuid: "c2f691b72da449edb2c24eff0dac1cdc" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 42223 } }
I20260924 16:17:09.676018 16699 leader_election.cc:288] T 25f519a0857e4f17993ad70a4caae2ef P c2f691b72da449edb2c24eff0dac1cdc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e2c01377a4f4d10877d8d4ab8297e2b (127.29.234.2:44079), c61dd0eaba9542068c71c4a81990d779 (127.29.234.3:38663)
I20260924 16:17:09.683103 16685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:09.676322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.1:56979 (local address 127.29.234.2:44079)
0924 16:17:09.676548 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:09.676555 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:09.677829 (+ 1274us) server_negotiation.cc:407] Connection header received
0924 16:17:09.678031 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:09.678035 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:09.678102 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:09.678200 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:17:09.678819 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:09.679492 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:09.681381 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:09.682169 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:09.682354 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:09.682427 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:09.682699 (+ 272us) server_negotiation.cc:299] Negotiation successful
0924 16:17:09.682826 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 16:17:09.683136 16686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:09.676407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.1:51529 (local address 127.29.234.3:38663)
0924 16:17:09.676609 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:09.676617 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:09.677714 (+ 1097us) server_negotiation.cc:407] Connection header received
0924 16:17:09.677936 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:09.677943 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:09.678010 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:09.678104 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:17:09.678686 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:09.679403 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:09.681473 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:09.682245 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:09.682437 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:09.682506 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:09.682779 (+ 273us) server_negotiation.cc:299] Negotiation successful
0924 16:17:09.682904 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 16:17:09.683267 16701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:09.676796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.2:44079 (local address 127.29.234.1:56979)
0924 16:17:09.677714 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0924 16:17:09.677728 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:17:09.677881 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:17:09.678220 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:17:09.678224 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:17:09.678263 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:17:09.678653 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:09.678662 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:09.679637 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:09.679642 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:09.681160 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:09.681174 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:09.682360 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:09.682366 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:09.682537 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:09.682564 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 16:17:09.682672 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 16:17:09.682814 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":820,"mutex_wait_us":41,"thread_start_us":487,"threads_started":1}
I20260924 16:17:09.683369 16700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:09.676760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.3:38663 (local address 127.29.234.1:51529)
0924 16:17:09.677612 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0924 16:17:09.677628 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:17:09.677789 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:17:09.678118 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:17:09.678122 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:17:09.678161 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:17:09.678534 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:09.678543 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:09.679524 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:09.679532 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:09.681263 (+ 1731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:09.681275 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:09.682443 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:09.682450 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:09.682623 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:09.682646 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 16:17:09.682751 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 16:17:09.682874 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":728,"mutex_wait_us":93,"thread_start_us":477,"threads_started":1}
I20260924 16:17:09.684593 16625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25f519a0857e4f17993ad70a4caae2ef" candidate_uuid: "c2f691b72da449edb2c24eff0dac1cdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c61dd0eaba9542068c71c4a81990d779" is_pre_election: true
I20260924 16:17:09.684593 16493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25f519a0857e4f17993ad70a4caae2ef" candidate_uuid: "c2f691b72da449edb2c24eff0dac1cdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2c01377a4f4d10877d8d4ab8297e2b" is_pre_election: true
I20260924 16:17:09.685196 16493 raft_consensus.cc:2454] T 25f519a0857e4f17993ad70a4caae2ef P 1e2c01377a4f4d10877d8d4ab8297e2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2f691b72da449edb2c24eff0dac1cdc in term 0.
I20260924 16:17:09.685196 16625 raft_consensus.cc:2454] T 25f519a0857e4f17993ad70a4caae2ef P c61dd0eaba9542068c71c4a81990d779 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2f691b72da449edb2c24eff0dac1cdc in term 0.
I20260924 16:17:09.685927 16292 leader_election.cc:302] T 25f519a0857e4f17993ad70a4caae2ef P c2f691b72da449edb2c24eff0dac1cdc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2f691b72da449edb2c24eff0dac1cdc, c61dd0eaba9542068c71c4a81990d779; no voters:
I20260924 16:17:09.686327 16699 raft_consensus.cc:2781] T 25f519a0857e4f17993ad70a4caae2ef P c2f691b72da449edb2c24eff0dac1cdc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:17:09.686496 16699 raft_consensus.cc:482] T 25f519a0857e4f17993ad70a4caae2ef P c2f691b72da449edb2c24eff0dac1cdc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:17:09.686605 16699 raft_consensus.cc:3037] T 25f519a0857e4f17993ad70a4caae2ef P c2f691b72da449edb2c24eff0dac1cdc [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:17:09.688880 16699 raft_consensus.cc:504] T 25f519a0857e4f17993ad70a4caae2ef P c2f691b72da449edb2c24eff0dac1cdc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2c01377a4f4d10877d8d4ab8297e2b" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 44079 } } peers { permanent_uuid: "c61dd0eaba9542068c71c4a81990d779" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 38663 } } peers { permanent_uuid: "c2f691b72da449edb2c24eff0dac1cdc" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 42223 } }
I20260924 16:17:09.689600 16699 leader_election.cc:288] T 25f519a0857e4f17993ad70a4caae2ef P c2f691b72da449edb2c24eff0dac1cdc [CANDIDATE]: Term 1 election: Requested vote from peers 1e2c01377a4f4d10877d8d4ab8297e2b (127.29.234.2:44079), c61dd0eaba9542068c71c4a81990d779 (127.29.234.3:38663)
I20260924 16:17:09.690138 16625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25f519a0857e4f17993ad70a4caae2ef" candidate_uuid: "c2f691b72da449edb2c24eff0dac1cdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c61dd0eaba9542068c71c4a81990d779"
I20260924 16:17:09.690146 16493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25f519a0857e4f17993ad70a4caae2ef" candidate_uuid: "c2f691b72da449edb2c24eff0dac1cdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e2c01377a4f4d10877d8d4ab8297e2b"
I20260924 16:17:09.690354 16625 raft_consensus.cc:3037] T 25f519a0857e4f17993ad70a4caae2ef P c61dd0eaba9542068c71c4a81990d779 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:17:09.690404 16493 raft_consensus.cc:3037] T 25f519a0857e4f17993ad70a4caae2ef P 1e2c01377a4f4d10877d8d4ab8297e2b [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:17:09.692603 16625 raft_consensus.cc:2454] T 25f519a0857e4f17993ad70a4caae2ef P c61dd0eaba9542068c71c4a81990d779 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2f691b72da449edb2c24eff0dac1cdc in term 1.
I20260924 16:17:09.692619 16493 raft_consensus.cc:2454] T 25f519a0857e4f17993ad70a4caae2ef P 1e2c01377a4f4d10877d8d4ab8297e2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2f691b72da449edb2c24eff0dac1cdc in term 1.
I20260924 16:17:09.693019 16292 leader_election.cc:302] T 25f519a0857e4f17993ad70a4caae2ef P c2f691b72da449edb2c24eff0dac1cdc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2f691b72da449edb2c24eff0dac1cdc, c61dd0eaba9542068c71c4a81990d779; no voters:
I20260924 16:17:09.693419 16699 raft_consensus.cc:2781] T 25f519a0857e4f17993ad70a4caae2ef P c2f691b72da449edb2c24eff0dac1cdc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:17:09.693840 16699 raft_consensus.cc:686] T 25f519a0857e4f17993ad70a4caae2ef P c2f691b72da449edb2c24eff0dac1cdc [term 1 LEADER]: Becoming Leader. State: Replica: c2f691b72da449edb2c24eff0dac1cdc, State: Running, Role: LEADER
I20260924 16:17:09.694329 16699 consensus_queue.cc:237] T 25f519a0857e4f17993ad70a4caae2ef P c2f691b72da449edb2c24eff0dac1cdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2c01377a4f4d10877d8d4ab8297e2b" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 44079 } } peers { permanent_uuid: "c61dd0eaba9542068c71c4a81990d779" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 38663 } } peers { permanent_uuid: "c2f691b72da449edb2c24eff0dac1cdc" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 42223 } }
I20260924 16:17:09.699091 16216 catalog_manager.cc:5495] T 25f519a0857e4f17993ad70a4caae2ef P c2f691b72da449edb2c24eff0dac1cdc reported cstate change: term changed from 0 to 1, leader changed from <none> to c2f691b72da449edb2c24eff0dac1cdc (127.29.234.1). New cstate: current_term: 1 leader_uuid: "c2f691b72da449edb2c24eff0dac1cdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e2c01377a4f4d10877d8d4ab8297e2b" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 44079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c61dd0eaba9542068c71c4a81990d779" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 38663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2f691b72da449edb2c24eff0dac1cdc" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 42223 } health_report { overall_health: HEALTHY } } }
I20260924 16:17:09.740965 16687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:09.735697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51862 (local address 127.29.234.1:42223)
0924 16:17:09.735909 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:09.735913 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:09.735930 (+ 17us) server_negotiation.cc:407] Connection header received
0924 16:17:09.736142 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:09.736145 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:09.736249 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:09.736373 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:17:09.736689 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:09.737453 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:09.738843 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:09.739233 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:09.739320 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:09.739761 (+ 441us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:17:09.740307 (+ 546us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:17:09.740413 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:09.740569 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 16:17:09.740636 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260924 16:17:09.749558 16625 raft_consensus.cc:1261] T 25f519a0857e4f17993ad70a4caae2ef P c61dd0eaba9542068c71c4a81990d779 [term 1 FOLLOWER]: Refusing update from remote peer c2f691b72da449edb2c24eff0dac1cdc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:17:09.749560 16493 raft_consensus.cc:1261] T 25f519a0857e4f17993ad70a4caae2ef P 1e2c01377a4f4d10877d8d4ab8297e2b [term 1 FOLLOWER]: Refusing update from remote peer c2f691b72da449edb2c24eff0dac1cdc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:17:09.750586 16702 consensus_queue.cc:1035] T 25f519a0857e4f17993ad70a4caae2ef P c2f691b72da449edb2c24eff0dac1cdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e2c01377a4f4d10877d8d4ab8297e2b" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 44079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:17:09.750926 16699 consensus_queue.cc:1035] T 25f519a0857e4f17993ad70a4caae2ef P c2f691b72da449edb2c24eff0dac1cdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c61dd0eaba9542068c71c4a81990d779" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 38663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:17:09.761970 16707 mvcc.cc:205] Tried to move back new op lower bound from 7332932115444400128 to 7332932115231858688. Current Snapshot: MvccSnapshot[applied={T|T < 7332932115444400128}]
I20260924 16:17:09.777071 16216 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59100:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:17:09.777350 16216 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59100:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:17:09.780985 16216 catalog_manager.cc:5782] T 00000000000000000000000000000000 P adccf10f80ca4dfe8d0fcc58653c71ad: Sending DeleteTablet for 3 replicas of tablet 25f519a0857e4f17993ad70a4caae2ef
I20260924 16:17:09.782251 16472 tablet_service.cc:1514] Processing DeleteTablet for tablet 25f519a0857e4f17993ad70a4caae2ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:17:09 UTC) from {username='slave'} at 127.0.0.1:49156
I20260924 16:17:09.782229 16604 tablet_service.cc:1514] Processing DeleteTablet for tablet 25f519a0857e4f17993ad70a4caae2ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:17:09 UTC) from {username='slave'} at 127.0.0.1:36272
I20260924 16:17:09.782485 16339 tablet_service.cc:1514] Processing DeleteTablet for tablet 25f519a0857e4f17993ad70a4caae2ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:17:09 UTC) from {username='slave'} at 127.0.0.1:51850
I20260924 16:17:09.784427 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 16276
I20260924 16:17:09.818919 16715 tablet_replica.cc:330] T 25f519a0857e4f17993ad70a4caae2ef P c61dd0eaba9542068c71c4a81990d779: stopping tablet replica
I20260924 16:17:09.819646 16714 tablet_replica.cc:330] T 25f519a0857e4f17993ad70a4caae2ef P 1e2c01377a4f4d10877d8d4ab8297e2b: stopping tablet replica
I20260924 16:17:09.820130 16714 raft_consensus.cc:2229] T 25f519a0857e4f17993ad70a4caae2ef P 1e2c01377a4f4d10877d8d4ab8297e2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:17:09.820659 16714 raft_consensus.cc:2258] T 25f519a0857e4f17993ad70a4caae2ef P 1e2c01377a4f4d10877d8d4ab8297e2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:17:09.821708 16715 raft_consensus.cc:2229] T 25f519a0857e4f17993ad70a4caae2ef P c61dd0eaba9542068c71c4a81990d779 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:17:09.822312 16715 raft_consensus.cc:2258] T 25f519a0857e4f17993ad70a4caae2ef P c61dd0eaba9542068c71c4a81990d779 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:17:09.825472 16716 tablet_replica.cc:330] T 25f519a0857e4f17993ad70a4caae2ef P c2f691b72da449edb2c24eff0dac1cdc: stopping tablet replica
I20260924 16:17:09.825986 16687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:09.787779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51874 (local address 127.29.234.1:42223)
0924 16:17:09.788015 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:09.788025 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:09.811145 (+ 23120us) server_negotiation.cc:407] Connection header received
0924 16:17:09.811806 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:09.811821 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:09.811974 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:09.812078 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:09.813065 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:09.814729 (+ 1664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:09.821278 (+ 6549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:09.822050 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:09.822183 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:09.823922 (+ 1739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:09.823938 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:09.823957 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:17:09.824856 (+ 899us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:09.824923 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:17:09.824958 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:09.825070 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:09.825429 (+ 359us) server_negotiation.cc:299] Negotiation successful
0924 16:17:09.825557 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 16:17:09.827219 16400 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:17:09.827720 16715 ts_tablet_manager.cc:1905] T 25f519a0857e4f17993ad70a4caae2ef P c61dd0eaba9542068c71c4a81990d779: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:17:09.828178 16714 ts_tablet_manager.cc:1905] T 25f519a0857e4f17993ad70a4caae2ef P 1e2c01377a4f4d10877d8d4ab8297e2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:17:09.833230 16714 ts_tablet_manager.cc:1918] T 25f519a0857e4f17993ad70a4caae2ef P 1e2c01377a4f4d10877d8d4ab8297e2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:17:09.833483 16714 log.cc:1196] T 25f519a0857e4f17993ad70a4caae2ef P 1e2c01377a4f4d10877d8d4ab8297e2b: Deleting WAL directory at /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266323169505-30632-0/minicluster-data/ts-1/wal/wals/25f519a0857e4f17993ad70a4caae2ef
I20260924 16:17:09.834192 16714 ts_tablet_manager.cc:1939] T 25f519a0857e4f17993ad70a4caae2ef P 1e2c01377a4f4d10877d8d4ab8297e2b: Deleting consensus metadata
I20260924 16:17:09.836354 16200 catalog_manager.cc:4841] TS 1e2c01377a4f4d10877d8d4ab8297e2b (127.29.234.2:44079): tablet 25f519a0857e4f17993ad70a4caae2ef (table test-table [id=190306af6eba4741bc7bd61f6487f20b]) successfully deleted
I20260924 16:17:09.837630 16715 ts_tablet_manager.cc:1918] T 25f519a0857e4f17993ad70a4caae2ef P c61dd0eaba9542068c71c4a81990d779: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:17:09.837812 16715 log.cc:1196] T 25f519a0857e4f17993ad70a4caae2ef P c61dd0eaba9542068c71c4a81990d779: Deleting WAL directory at /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790266323169505-30632-0/minicluster-data/ts-2/wal/wals/25f519a0857e4f17993ad70a4caae2ef
I20260924 16:17:09.838389 16715 ts_tablet_manager.cc:1939] T 25f519a0857e4f17993ad70a4caae2ef P c61dd0eaba9542068c71c4a81990d779: Deleting consensus metadata
I20260924 16:17:09.840363 16201 catalog_manager.cc:4841] TS c61dd0eaba9542068c71c4a81990d779 (127.29.234.3:38663): tablet 25f519a0857e4f17993ad70a4caae2ef (table test-table [id=190306af6eba4741bc7bd61f6487f20b]) successfully deleted
I20260924 16:17:09.829440 16716 raft_consensus.cc:2229] T 25f519a0857e4f17993ad70a4caae2ef P c2f691b72da449edb2c24eff0dac1cdc [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:17:09.933476 16716 raft_consensus.cc:2258] T 25f519a0857e4f17993ad70a4caae2ef P c2f691b72da449edb2c24eff0dac1cdc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:17:09.933549 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 16276
I20260924 16:17:09.953859 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 16409
W20260924 16:17:09.954195 16200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.234.1:42223 (error 108)
I20260924 16:17:09.956110 16689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:09.955668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:42223 (local address 127.0.0.1:51878)
0924 16:17:09.955864 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0924 16:17:09.955977 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.1:42223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260924 16:17:09.956383 16200 catalog_manager.cc:4584] TS c2f691b72da449edb2c24eff0dac1cdc (127.29.234.1:42223): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 25f519a0857e4f17993ad70a4caae2ef: Network error: Client connection negotiation failed: client connection to 127.29.234.1:42223: connect: Connection refused (error 111)
I20260924 16:17:09.961301 16685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:09.954808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49170 (local address 127.29.234.2:44079)
0924 16:17:09.954981 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:09.954986 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:09.955183 (+ 197us) server_negotiation.cc:407] Connection header received
0924 16:17:09.955404 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:09.955410 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:09.955505 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:09.955625 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:09.956046 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:09.956966 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:09.958311 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:09.958726 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:09.958872 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:09.959916 (+ 1044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:09.959927 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:09.959942 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:17:09.960538 (+ 596us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:09.960585 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:17:09.960606 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:09.960736 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:09.960871 (+ 135us) server_negotiation.cc:299] Negotiation successful
0924 16:17:09.960982 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 16:17:09.962081 16533 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:17:10.010035 16689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:10.009474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:42223 (local address 127.0.0.1:51892)
0924 16:17:10.009711 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0924 16:17:10.009825 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.1:42223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260924 16:17:10.011220 16689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:10.010753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:42223 (local address 127.0.0.1:51904)
0924 16:17:10.010914 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 16:17:10.011006 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.1:42223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260924 16:17:10.052546 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 16409
I20260924 16:17:10.070314 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 16541
I20260924 16:17:10.071973 16689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:10.071497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:42223 (local address 127.0.0.1:51910)
0924 16:17:10.071679 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 16:17:10.071801 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.1:42223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260924 16:17:10.073050 16689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:10.072711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:42223 (local address 127.0.0.1:51912)
0924 16:17:10.072873 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 16:17:10.072939 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.1:42223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260924 16:17:10.077673 16686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:10.071186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36282 (local address 127.29.234.3:38663)
0924 16:17:10.071352 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:10.071356 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:10.071613 (+ 257us) server_negotiation.cc:407] Connection header received
0924 16:17:10.071875 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:10.071880 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:10.072008 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:10.072124 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:10.072749 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:10.073678 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:10.074982 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:10.075366 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:10.075458 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:10.076419 (+ 961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:10.076428 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:10.076447 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:17:10.077041 (+ 594us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:10.077089 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:17:10.077108 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:10.077196 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:10.077282 (+ 86us) server_negotiation.cc:299] Negotiation successful
0924 16:17:10.077436 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 16:17:10.078343 16666 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:17:10.153957 16689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:10.153382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:42223 (local address 127.0.0.1:51928)
0924 16:17:10.153618 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0924 16:17:10.153773 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.1:42223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260924 16:17:10.155097 16689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:10.154676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:42223 (local address 127.0.0.1:51936)
0924 16:17:10.154892 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0924 16:17:10.154979 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.1:42223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260924 16:17:10.166666 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 16541
I20260924 16:17:10.185024 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 16184
I20260924 16:17:10.186208 16246 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:17:10.295687 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 16184
2026-09-24T16:17:10Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2747 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:17:10 dist-test-slave-v5j9 krb5kdc[16725](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:17:10 dist-test-slave-v5j9 krb5kdc[16725](info): set up 2 sockets
Sep 24 16:17:10 dist-test-slave-v5j9 krb5kdc[16725](info): commencing operation
krb5kdc: starting...
W20260924 16:17:11.416945 30632 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Sep 24 16:17:11 dist-test-slave-v5j9 krb5kdc[16725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:17:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:17:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.234.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab.
Entry for principal kudu/127.29.234.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab.
I20260924 16:17:11.678200 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:33227
--webserver_interface=127.29.234.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab
--principal=kudu/127.29.234.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:43179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.234.62:33227
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:17:12.033690 16738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:17:12.033995 16738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:17:12.034164 16738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:17:12.041486 16738 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:17:12.041594 16738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:17:12.041720 16738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:17:12.041821 16738 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:17:12.041914 16738 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:17:12.051949 16738 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:43179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.234.62:33227
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab
--principal=kudu/127.29.234.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:33227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.234.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:17:12.053378 16738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:17:12.054715 16738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:17:12.067057 16744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:12.067137 16746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:12.067062 16743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:17:12.067955 16738 server_base.cc:1034] running on GCE node
I20260924 16:17:12.068835 16738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:17:12.070405 16738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:17:12.071661 16738 hybrid_clock.cc:648] HybridClock initialized: now 1790266632071613 us; error 50 us; skew 500 ppm
I20260924 16:17:12.072028 16738 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:17:12 dist-test-slave-v5j9 krb5kdc[16725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266632, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:17:12.079581 16738 init.cc:377] Logged in from keytab as kudu/127.29.234.62@KRBTEST.COM (short username kudu)
I20260924 16:17:12.081178 16738 webserver.cc:460] Webserver started at http://127.29.234.62:33869/ using document root <none> and password file <none>
I20260924 16:17:12.082000 16738 fs_manager.cc:359] Metadata directory not provided
I20260924 16:17:12.082131 16738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:17:12.082475 16738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:17:12.085506 16738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/minicluster-data/master-0/data/instance:
uuid: "34b6818dc7ab4bc2a81644561a3c67f3"
format_stamp: "Formatted at 2026-09-24 16:17:12 on dist-test-slave-v5j9"
server_key: "ff796061d3c9d2de58ecd3ba6bf8fe21"
server_key_iv: "e2be592cdf4be23c86f5f9b1fe3d6cfe"
server_key_version: "encryptionkey@0"
I20260924 16:17:12.086283 16738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/minicluster-data/master-0/wal/instance:
uuid: "34b6818dc7ab4bc2a81644561a3c67f3"
format_stamp: "Formatted at 2026-09-24 16:17:12 on dist-test-slave-v5j9"
server_key: "ff796061d3c9d2de58ecd3ba6bf8fe21"
server_key_iv: "e2be592cdf4be23c86f5f9b1fe3d6cfe"
server_key_version: "encryptionkey@0"
I20260924 16:17:12.091467 16738 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 16:17:12.095064 16753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:17:12.096390 16738 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 16:17:12.096529 16738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/minicluster-data/master-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/minicluster-data/master-0/wal
uuid: "34b6818dc7ab4bc2a81644561a3c67f3"
format_stamp: "Formatted at 2026-09-24 16:17:12 on dist-test-slave-v5j9"
server_key: "ff796061d3c9d2de58ecd3ba6bf8fe21"
server_key_iv: "e2be592cdf4be23c86f5f9b1fe3d6cfe"
server_key_version: "encryptionkey@0"
I20260924 16:17:12.096700 16738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:17:12.110309 16738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:17:12.114434 16738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:17:12.114671 16738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:17:12.133543 16738 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.62:33227
I20260924 16:17:12.133539 16806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.62:33227 every 8 connection(s)
I20260924 16:17:12.134949 16738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
I20260924 16:17:12.138829 16807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:17:12.142819 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 16738
I20260924 16:17:12.143023 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/minicluster-data/master-0/wal/instance
I20260924 16:17:12.143602 30632 external_mini_cluster.cc:1428] Setting key d5534a4bf9e3f8f472c6f99041d2d40b
I20260924 16:17:12.152465 16807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34b6818dc7ab4bc2a81644561a3c67f3: Bootstrap starting.
I20260924 16:17:12.156450 16807 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34b6818dc7ab4bc2a81644561a3c67f3: Neither blocks nor log segments found. Creating new log.
I20260924 16:17:12.157862 16807 log.cc:824] T 00000000000000000000000000000000 P 34b6818dc7ab4bc2a81644561a3c67f3: Log is configured to *not* fsync() on all Append() calls
Sep 24 16:17:12 dist-test-slave-v5j9 krb5kdc[16725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:17:12.161047 16807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34b6818dc7ab4bc2a81644561a3c67f3: No bootstrap required, opened a new log
I20260924 16:17:12.167552 16810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:12.145721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44448 (local address 127.29.234.62:33227)
0924 16:17:12.146557 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:12.146570 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:12.146613 (+ 43us) server_negotiation.cc:407] Connection header received
0924 16:17:12.148262 (+ 1649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:12.148285 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:12.148714 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:12.149056 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:12.150934 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:12.152222 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:12.154110 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:12.154780 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:12.154924 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:12.160164 (+ 5240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:12.160172 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:12.160187 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:17:12.160225 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:12.163430 (+ 3205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:17:12.164242 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:17:12.164248 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:17:12.164255 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:17:12.164421 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:17:12.164845 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:17:12.164848 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:17:12.164850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:17:12.165393 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:12.165629 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:12.166131 (+ 502us) server_negotiation.cc:299] Negotiation successful
0924 16:17:12.166487 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":380,"threads_started":1}
I20260924 16:17:12.170498 16807 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34b6818dc7ab4bc2a81644561a3c67f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b6818dc7ab4bc2a81644561a3c67f3" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 33227 } }
I20260924 16:17:12.170923 16807 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34b6818dc7ab4bc2a81644561a3c67f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:17:12.171030 16807 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34b6818dc7ab4bc2a81644561a3c67f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34b6818dc7ab4bc2a81644561a3c67f3, State: Initialized, Role: FOLLOWER
I20260924 16:17:12.171633 16807 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34b6818dc7ab4bc2a81644561a3c67f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b6818dc7ab4bc2a81644561a3c67f3" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 33227 } }
I20260924 16:17:12.171867 16807 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34b6818dc7ab4bc2a81644561a3c67f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:17:12.172006 16807 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34b6818dc7ab4bc2a81644561a3c67f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:17:12.172192 16807 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34b6818dc7ab4bc2a81644561a3c67f3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:17:12.174585 16807 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34b6818dc7ab4bc2a81644561a3c67f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b6818dc7ab4bc2a81644561a3c67f3" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 33227 } }
I20260924 16:17:12.175087 16807 leader_election.cc:302] T 00000000000000000000000000000000 P 34b6818dc7ab4bc2a81644561a3c67f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34b6818dc7ab4bc2a81644561a3c67f3; no voters:
I20260924 16:17:12.175654 16807 leader_election.cc:288] T 00000000000000000000000000000000 P 34b6818dc7ab4bc2a81644561a3c67f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:17:12.175776 16812 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34b6818dc7ab4bc2a81644561a3c67f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:17:12.176481 16812 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34b6818dc7ab4bc2a81644561a3c67f3 [term 1 LEADER]: Becoming Leader. State: Replica: 34b6818dc7ab4bc2a81644561a3c67f3, State: Running, Role: LEADER
I20260924 16:17:12.177031 16812 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34b6818dc7ab4bc2a81644561a3c67f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b6818dc7ab4bc2a81644561a3c67f3" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 33227 } }
I20260924 16:17:12.177945 16807 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34b6818dc7ab4bc2a81644561a3c67f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:17:12.180241 16814 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34b6818dc7ab4bc2a81644561a3c67f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34b6818dc7ab4bc2a81644561a3c67f3. Latest consensus state: current_term: 1 leader_uuid: "34b6818dc7ab4bc2a81644561a3c67f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b6818dc7ab4bc2a81644561a3c67f3" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 33227 } } }
I20260924 16:17:12.180267 16813 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34b6818dc7ab4bc2a81644561a3c67f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34b6818dc7ab4bc2a81644561a3c67f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b6818dc7ab4bc2a81644561a3c67f3" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 33227 } } }
I20260924 16:17:12.180904 16814 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34b6818dc7ab4bc2a81644561a3c67f3 [sys.catalog]: This master's current role is: LEADER
I20260924 16:17:12.180908 16813 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34b6818dc7ab4bc2a81644561a3c67f3 [sys.catalog]: This master's current role is: LEADER
I20260924 16:17:12.181830 16821 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:17:12.187542 16821 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:17:12.196409 16821 catalog_manager.cc:1347] Generated new cluster ID: 4b0b0e593173479893c8114226f30f0d
I20260924 16:17:12.196537 16821 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:17:12.207542 16821 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:17:12.209458 16821 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:17:12.222586 16821 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34b6818dc7ab4bc2a81644561a3c67f3: Generated new TSK 0
I20260924 16:17:12.223588 16821 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.234.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab.
Entry for principal kudu/127.29.234.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab.
I20260924 16:17:12.284631 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.1:0
--local_ip_for_outbound_sockets=127.29.234.1
--webserver_interface=127.29.234.1
--webserver_port=0
--tserver_master_addrs=127.29.234.62:33227
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab
--principal=kudu/127.29.234.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:43179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:17:12.632596 16833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:17:12.632858 16833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:17:12.633023 16833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:17:12.640004 16833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:17:12.640129 16833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:17:12.640321 16833 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.1
I20260924 16:17:12.649698 16833 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:43179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab
--principal=kudu/127.29.234.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.234.1
--webserver_port=0
--tserver_master_addrs=127.29.234.62:33227
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.1
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:17:12.650936 16833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:17:12.652310 16833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:17:12.665869 16840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:12.665841 16838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:12.665920 16842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:17:12.666544 16833 server_base.cc:1034] running on GCE node
I20260924 16:17:12.667354 16833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:17:12.668390 16833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:17:12.669589 16833 hybrid_clock.cc:648] HybridClock initialized: now 1790266632669535 us; error 45 us; skew 500 ppm
I20260924 16:17:12.669883 16833 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:17:12 dist-test-slave-v5j9 krb5kdc[16725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266632, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:17:12.672981 16833 init.cc:377] Logged in from keytab as kudu/127.29.234.1@KRBTEST.COM (short username kudu)
I20260924 16:17:12.674453 16833 webserver.cc:460] Webserver started at http://127.29.234.1:45097/ using document root <none> and password file <none>
I20260924 16:17:12.675155 16833 fs_manager.cc:359] Metadata directory not provided
I20260924 16:17:12.675304 16833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:17:12.675738 16833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:17:12.679369 16833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/minicluster-data/ts-0/data/instance:
uuid: "d1e8d44dcc5f473aa2ab08cffc94a2eb"
format_stamp: "Formatted at 2026-09-24 16:17:12 on dist-test-slave-v5j9"
server_key: "5f80e9ad1c81eaf42504fb21420a1160"
server_key_iv: "489c325f28bdf8523772f7a14a1c05c3"
server_key_version: "encryptionkey@0"
I20260924 16:17:12.680117 16833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/minicluster-data/ts-0/wal/instance:
uuid: "d1e8d44dcc5f473aa2ab08cffc94a2eb"
format_stamp: "Formatted at 2026-09-24 16:17:12 on dist-test-slave-v5j9"
server_key: "5f80e9ad1c81eaf42504fb21420a1160"
server_key_iv: "489c325f28bdf8523772f7a14a1c05c3"
server_key_version: "encryptionkey@0"
I20260924 16:17:12.685482 16833 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260924 16:17:12.688830 16849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:17:12.690209 16833 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:17:12.690469 16833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/minicluster-data/ts-0/wal
uuid: "d1e8d44dcc5f473aa2ab08cffc94a2eb"
format_stamp: "Formatted at 2026-09-24 16:17:12 on dist-test-slave-v5j9"
server_key: "5f80e9ad1c81eaf42504fb21420a1160"
server_key_iv: "489c325f28bdf8523772f7a14a1c05c3"
server_key_version: "encryptionkey@0"
I20260924 16:17:12.690716 16833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:17:12.706756 16833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:17:12.710626 16833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:17:12.710875 16833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:17:12.711887 16833 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:17:12.713172 16833 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:17:12.713244 16833 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:17:12.713330 16833 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:17:12.713431 16833 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260924 16:17:12.746920 16833 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.1:33341
I20260924 16:17:12.746970 16962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.1:33341 every 8 connection(s)
I20260924 16:17:12.748245 16833 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
I20260924 16:17:12.752161 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 16833
I20260924 16:17:12.752382 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/minicluster-data/ts-0/wal/instance
I20260924 16:17:12.752983 30632 external_mini_cluster.cc:1428] Setting key 75aac38736abc0de0f2ed10b68203b4a
Sep 24 16:17:12 dist-test-slave-v5j9 krb5kdc[16725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266632, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.1@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:17:12.770478 16965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:12.751030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.1:48059 (local address 127.29.234.62:33227)
0924 16:17:12.751505 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:12.751509 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:12.752614 (+ 1105us) server_negotiation.cc:407] Connection header received
0924 16:17:12.754730 (+ 2116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:12.754735 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:12.754850 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:12.754965 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:12.756513 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:12.757240 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:12.758842 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:12.759249 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:12.759390 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:12.763115 (+ 3725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:12.763121 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:12.763126 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:17:12.763170 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:12.766896 (+ 3726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:17:12.767662 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:17:12.767672 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:17:12.767679 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:17:12.767848 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:17:12.768276 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:17:12.768285 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:17:12.768292 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:17:12.768715 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:12.768876 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:12.769653 (+ 777us) server_negotiation.cc:299] Negotiation successful
0924 16:17:12.769939 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":268,"threads_started":1}
I20260924 16:17:12.771093 16966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:12.751562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:33227 (local address 127.29.234.1:48059)
0924 16:17:12.752419 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0924 16:17:12.752497 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0924 16:17:12.754457 (+ 1960us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:17:12.755218 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:17:12.755226 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:17:12.755750 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:17:12.756336 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:12.756359 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:12.757426 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:12.757436 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:12.758688 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:12.758700 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:12.759486 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:12.759495 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:12.759744 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:12.760444 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:17:12.760466 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:17:12.762869 (+ 2403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:17:12.767046 (+ 4177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:17:12.767053 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:17:12.767067 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:17:12.767506 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:17:12.767970 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:17:12.767973 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:17:12.767975 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:17:12.768151 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:17:12.768847 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:17:12.768853 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:17:12.769180 (+ 327us) client_negotiation.cc:769] Sending connection context
0924 16:17:12.769479 (+ 299us) client_negotiation.cc:240] Negotiation successful
0924 16:17:12.769882 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":322,"threads_started":1}
I20260924 16:17:12.772876 16963 heartbeater.cc:347] Connected to a master server at 127.29.234.62:33227
I20260924 16:17:12.773197 16963 heartbeater.cc:464] Registering TS with master...
I20260924 16:17:12.774019 16963 heartbeater.cc:511] Master 127.29.234.62:33227 requested a full tablet report, sending...
I20260924 16:17:12.776482 16771 ts_manager.cc:194] Registered new tserver with Master: d1e8d44dcc5f473aa2ab08cffc94a2eb (127.29.234.1:33341)
I20260924 16:17:12.779122 16771 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.234.1@KRBTEST.COM'} at 127.29.234.1:48059
WARNING: no policy specified for kudu/127.29.234.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab.
Entry for principal kudu/127.29.234.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab.
I20260924 16:17:12.814631 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.2:0
--local_ip_for_outbound_sockets=127.29.234.2
--webserver_interface=127.29.234.2
--webserver_port=0
--tserver_master_addrs=127.29.234.62:33227
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab
--principal=kudu/127.29.234.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:43179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:17:13.147570 16969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:17:13.147868 16969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:17:13.148066 16969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:17:13.155354 16969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:17:13.155463 16969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:17:13.155625 16969 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.2
I20260924 16:17:13.165701 16969 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:43179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab
--principal=kudu/127.29.234.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.234.2
--webserver_port=0
--tserver_master_addrs=127.29.234.62:33227
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.2
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:17:13.167090 16969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:17:13.168617 16969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:17:13.182624 16975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:13.182719 16977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:13.182700 16974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:17:13.183283 16969 server_base.cc:1034] running on GCE node
I20260924 16:17:13.183941 16969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:17:13.184887 16969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:17:13.186093 16969 hybrid_clock.cc:648] HybridClock initialized: now 1790266633186019 us; error 53 us; skew 500 ppm
I20260924 16:17:13.186419 16969 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:17:13 dist-test-slave-v5j9 krb5kdc[16725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266633, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:17:13.189591 16969 init.cc:377] Logged in from keytab as kudu/127.29.234.2@KRBTEST.COM (short username kudu)
I20260924 16:17:13.190997 16969 webserver.cc:460] Webserver started at http://127.29.234.2:44711/ using document root <none> and password file <none>
I20260924 16:17:13.191696 16969 fs_manager.cc:359] Metadata directory not provided
I20260924 16:17:13.191846 16969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:17:13.192163 16969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:17:13.195254 16969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/minicluster-data/ts-1/data/instance:
uuid: "99343d3fd9f047d392bc7fdbb500a059"
format_stamp: "Formatted at 2026-09-24 16:17:13 on dist-test-slave-v5j9"
server_key: "ab4be6a97f56abf0cefb47a47ddd6621"
server_key_iv: "8322a3458850bf6a9dcfcbd81b3e9015"
server_key_version: "encryptionkey@0"
I20260924 16:17:13.196022 16969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/minicluster-data/ts-1/wal/instance:
uuid: "99343d3fd9f047d392bc7fdbb500a059"
format_stamp: "Formatted at 2026-09-24 16:17:13 on dist-test-slave-v5j9"
server_key: "ab4be6a97f56abf0cefb47a47ddd6621"
server_key_iv: "8322a3458850bf6a9dcfcbd81b3e9015"
server_key_version: "encryptionkey@0"
I20260924 16:17:13.201254 16969 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 16:17:13.204578 16985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:17:13.205912 16969 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:17:13.206095 16969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/minicluster-data/ts-1/wal
uuid: "99343d3fd9f047d392bc7fdbb500a059"
format_stamp: "Formatted at 2026-09-24 16:17:13 on dist-test-slave-v5j9"
server_key: "ab4be6a97f56abf0cefb47a47ddd6621"
server_key_iv: "8322a3458850bf6a9dcfcbd81b3e9015"
server_key_version: "encryptionkey@0"
I20260924 16:17:13.206302 16969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:17:13.225503 16969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:17:13.229897 16969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:17:13.230193 16969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:17:13.231155 16969 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:17:13.232529 16969 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:17:13.232602 16969 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:17:13.232755 16969 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:17:13.232821 16969 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:17:13.264283 16969 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.2:36293
I20260924 16:17:13.264324 17098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.2:36293 every 8 connection(s)
I20260924 16:17:13.265686 16969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
I20260924 16:17:13.270001 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 16969
I20260924 16:17:13.270241 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/minicluster-data/ts-1/wal/instance
I20260924 16:17:13.270833 30632 external_mini_cluster.cc:1428] Setting key 8161cc83557c81dae4d16d8e57f74c0b
Sep 24 16:17:13 dist-test-slave-v5j9 krb5kdc[16725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266633, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.2@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:17:13.290004 16965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:13.268359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.2:33039 (local address 127.29.234.62:33227)
0924 16:17:13.268543 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:13.268548 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:13.269936 (+ 1388us) server_negotiation.cc:407] Connection header received
0924 16:17:13.272381 (+ 2445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:13.272387 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:13.272463 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:13.272571 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:13.274716 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:13.275429 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:13.277928 (+ 2499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:13.278471 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:13.278634 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:13.282950 (+ 4316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:13.282959 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:13.282965 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:17:13.283014 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:13.286660 (+ 3646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:17:13.287404 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:17:13.287412 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:17:13.287418 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:17:13.287582 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:17:13.287984 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:17:13.287991 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:17:13.287997 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:17:13.288376 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:13.288524 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:13.289257 (+ 733us) server_negotiation.cc:299] Negotiation successful
0924 16:17:13.289554 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 16:17:13.290787 17101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:13.268911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:33227 (local address 127.29.234.2:33039)
0924 16:17:13.269739 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0924 16:17:13.269809 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0924 16:17:13.272118 (+ 2309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:17:13.272866 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:17:13.272897 (+ 31us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:17:13.273670 (+ 773us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:17:13.274511 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:13.274539 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:13.275578 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:13.275586 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:13.277737 (+ 2151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:13.277755 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:13.278719 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:13.278729 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:13.279078 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:13.279879 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:17:13.279918 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:17:13.282682 (+ 2764us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:17:13.286811 (+ 4129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:17:13.286818 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:17:13.286832 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:17:13.287262 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:17:13.287696 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:17:13.287699 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:17:13.287701 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:17:13.287868 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:17:13.288507 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:17:13.288513 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:17:13.288869 (+ 356us) client_negotiation.cc:769] Sending connection context
0924 16:17:13.289100 (+ 231us) client_negotiation.cc:240] Negotiation successful
0924 16:17:13.289598 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":256,"threads_started":1}
I20260924 16:17:13.292418 17099 heartbeater.cc:347] Connected to a master server at 127.29.234.62:33227
I20260924 16:17:13.292771 17099 heartbeater.cc:464] Registering TS with master...
I20260924 16:17:13.293774 17099 heartbeater.cc:511] Master 127.29.234.62:33227 requested a full tablet report, sending...
I20260924 16:17:13.295605 16771 ts_manager.cc:194] Registered new tserver with Master: 99343d3fd9f047d392bc7fdbb500a059 (127.29.234.2:36293)
I20260924 16:17:13.296469 16771 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.234.2@KRBTEST.COM'} at 127.29.234.2:33039
WARNING: no policy specified for kudu/127.29.234.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab.
Entry for principal kudu/127.29.234.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab.
I20260924 16:17:13.333264 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.3:0
--local_ip_for_outbound_sockets=127.29.234.3
--webserver_interface=127.29.234.3
--webserver_port=0
--tserver_master_addrs=127.29.234.62:33227
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab
--principal=kudu/127.29.234.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:43179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:17:13.667689 17104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:17:13.667929 17104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:17:13.668118 17104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:17:13.675061 17104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:17:13.675153 17104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:17:13.675302 17104 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.3
I20260924 16:17:13.684924 17104 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:43179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab
--principal=kudu/127.29.234.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.234.3
--webserver_port=0
--tserver_master_addrs=127.29.234.62:33227
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.3
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:17:13.686355 17104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:17:13.687606 17104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:17:13.700750 17110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:13.700696 17109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:13.701278 17112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:17:13.701457 17104 server_base.cc:1034] running on GCE node
I20260924 16:17:13.702008 17104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:17:13.702839 17104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:17:13.704063 17104 hybrid_clock.cc:648] HybridClock initialized: now 1790266633704006 us; error 41 us; skew 500 ppm
I20260924 16:17:13.704411 17104 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:17:13 dist-test-slave-v5j9 krb5kdc[16725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266633, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:17:13.707481 17104 init.cc:377] Logged in from keytab as kudu/127.29.234.3@KRBTEST.COM (short username kudu)
I20260924 16:17:13.708803 17104 webserver.cc:460] Webserver started at http://127.29.234.3:40833/ using document root <none> and password file <none>
I20260924 16:17:13.709604 17104 fs_manager.cc:359] Metadata directory not provided
I20260924 16:17:13.709731 17104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:17:13.710047 17104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:17:13.713053 17104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/minicluster-data/ts-2/data/instance:
uuid: "8ec37953e2c24727a41e532cdad10e62"
format_stamp: "Formatted at 2026-09-24 16:17:13 on dist-test-slave-v5j9"
server_key: "dcf510a657091b3488f035581c9084fc"
server_key_iv: "ba06ed1e3405e66b058a0d3fd93e6abb"
server_key_version: "encryptionkey@0"
I20260924 16:17:13.713866 17104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/minicluster-data/ts-2/wal/instance:
uuid: "8ec37953e2c24727a41e532cdad10e62"
format_stamp: "Formatted at 2026-09-24 16:17:13 on dist-test-slave-v5j9"
server_key: "dcf510a657091b3488f035581c9084fc"
server_key_iv: "ba06ed1e3405e66b058a0d3fd93e6abb"
server_key_version: "encryptionkey@0"
I20260924 16:17:13.719054 17104 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 16:17:13.722923 17119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:17:13.724284 17104 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:17:13.724476 17104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/minicluster-data/ts-2/wal
uuid: "8ec37953e2c24727a41e532cdad10e62"
format_stamp: "Formatted at 2026-09-24 16:17:13 on dist-test-slave-v5j9"
server_key: "dcf510a657091b3488f035581c9084fc"
server_key_iv: "ba06ed1e3405e66b058a0d3fd93e6abb"
server_key_version: "encryptionkey@0"
I20260924 16:17:13.724684 17104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:17:13.766340 17104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:17:13.770407 17104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:17:13.770712 17104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:17:13.771723 17104 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:17:13.773072 17104 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:17:13.773177 17104 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:17:13.773340 17104 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:17:13.773439 17104 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260924 16:17:13.782469 16963 heartbeater.cc:503] Master 127.29.234.62:33227 was elected leader, sending a full tablet report...
I20260924 16:17:13.807428 17104 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.3:46247
I20260924 16:17:13.807466 17233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.3:46247 every 8 connection(s)
I20260924 16:17:13.808804 17104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
I20260924 16:17:13.818035 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 17104
I20260924 16:17:13.818228 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/minicluster-data/ts-2/wal/instance
I20260924 16:17:13.818677 30632 external_mini_cluster.cc:1428] Setting key f6df3a8c7d23311ea2da1f7236baaed6
Sep 24 16:17:13 dist-test-slave-v5j9 krb5kdc[16725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266633, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.3@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:17:13.828976 17236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:13.811527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.3:34571 (local address 127.29.234.62:33227)
0924 16:17:13.811965 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:13.811972 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:13.813069 (+ 1097us) server_negotiation.cc:407] Connection header received
0924 16:17:13.814889 (+ 1820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:13.814893 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:13.814960 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:13.815056 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:13.816477 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:13.817251 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:13.819071 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:13.819505 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:13.819669 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:13.823784 (+ 4115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:13.823789 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:13.823792 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:17:13.823824 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:13.826254 (+ 2430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:17:13.826904 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:17:13.826908 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:17:13.826910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:17:13.827025 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:17:13.827387 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:17:13.827390 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:17:13.827392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:17:13.827643 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:13.827801 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:13.828459 (+ 658us) server_negotiation.cc:299] Negotiation successful
0924 16:17:13.828659 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":208,"threads_started":1}
I20260924 16:17:13.829939 17237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:13.812193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:33227 (local address 127.29.234.3:34571)
0924 16:17:13.812919 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0924 16:17:13.812956 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 16:17:13.814624 (+ 1668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:17:13.815265 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:17:13.815272 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:17:13.815776 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:17:13.816332 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:13.816348 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:13.817439 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:13.817443 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:13.818923 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:13.818939 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:13.819756 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:13.819761 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:13.820247 (+ 486us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:13.820986 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:17:13.821031 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:17:13.823586 (+ 2555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:17:13.826374 (+ 2788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:17:13.826380 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:17:13.826393 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:17:13.826797 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:17:13.827114 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:17:13.827117 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:17:13.827119 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:17:13.827290 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:17:13.827765 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:17:13.827771 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:17:13.828064 (+ 293us) client_negotiation.cc:769] Sending connection context
0924 16:17:13.828297 (+ 233us) client_negotiation.cc:240] Negotiation successful
0924 16:17:13.828741 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":290,"threads_started":1}
I20260924 16:17:13.831274 17234 heartbeater.cc:347] Connected to a master server at 127.29.234.62:33227
I20260924 16:17:13.831576 17234 heartbeater.cc:464] Registering TS with master...
I20260924 16:17:13.832284 17234 heartbeater.cc:511] Master 127.29.234.62:33227 requested a full tablet report, sending...
I20260924 16:17:13.833989 16771 ts_manager.cc:194] Registered new tserver with Master: 8ec37953e2c24727a41e532cdad10e62 (127.29.234.3:46247)
I20260924 16:17:13.834599 30632 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:17:13.835078 16771 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.234.3@KRBTEST.COM'} at 127.29.234.3:34571
I20260924 16:17:13.855914 17236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:13.842985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35584 (local address 127.29.234.62:33227)
0924 16:17:13.843166 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:13.843174 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:13.843885 (+ 711us) server_negotiation.cc:407] Connection header received
0924 16:17:13.844072 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:13.844077 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:13.844152 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:13.844263 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:13.845753 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:13.846475 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:13.847771 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:13.848174 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:13.848302 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:13.850906 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:13.850913 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:13.850917 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:17:13.850957 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:13.853187 (+ 2230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:17:13.853941 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:17:13.853946 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:17:13.853950 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:17:13.854072 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:17:13.854525 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:17:13.854530 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:17:13.854535 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:17:13.854823 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:13.855088 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:13.855476 (+ 388us) server_negotiation.cc:299] Negotiation successful
0924 16:17:13.855661 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 16:17:13.859543 16771 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35584:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:17:13.866452 16771 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:17:13.895205 17247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:13.879163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:33341 (local address 127.0.0.1:52970)
0924 16:17:13.880528 (+ 1365us) negotiation.cc:107] Waiting for socket to connect
0924 16:17:13.880579 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 16:17:13.880744 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:17:13.882640 (+ 1896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:17:13.882652 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:17:13.882730 (+ 78us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:17:13.883345 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:13.883363 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:13.887656 (+ 4293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:13.887664 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:13.891019 (+ 3355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:13.891042 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:13.893806 (+ 2764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:13.893817 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:13.894120 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:13.894201 (+ 81us) client_negotiation.cc:769] Sending connection context
0924 16:17:13.894409 (+ 208us) client_negotiation.cc:240] Negotiation successful
0924 16:17:13.894647 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1147,"mutex_wait_us":21,"thread_start_us":962,"threads_started":1}
I20260924 16:17:13.895915 17250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:13.879906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52970 (local address 127.29.234.1:33341)
0924 16:17:13.881449 (+ 1543us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:13.881458 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:13.881485 (+ 27us) server_negotiation.cc:407] Connection header received
0924 16:17:13.881596 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:13.881607 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:13.881948 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:13.883573 (+ 1625us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:17:13.883901 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:13.886053 (+ 2152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:13.891314 (+ 5261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:13.893473 (+ 2159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:13.894072 (+ 599us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:13.895305 (+ 1233us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:13.895385 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 16:17:13.895566 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1301,"thread_start_us":1123,"threads_started":1}
I20260924 16:17:13.899116 17249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:13.879223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.2:36293 (local address 127.0.0.1:37880)
0924 16:17:13.887273 (+ 8050us) negotiation.cc:107] Waiting for socket to connect
0924 16:17:13.887304 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 16:17:13.887459 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:17:13.888536 (+ 1077us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:17:13.888545 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:17:13.888587 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:17:13.889164 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:13.889179 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:13.891777 (+ 2598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:13.891785 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:13.896255 (+ 4470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:13.896271 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:13.898187 (+ 1916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:13.898195 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:13.898384 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:13.898423 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 16:17:13.898498 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 16:17:13.898664 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7894,"mutex_wait_us":30,"thread_start_us":7698,"threads_started":1}
I20260924 16:17:13.900367 17248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:13.879229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37880 (local address 127.29.234.2:36293)
0924 16:17:13.886743 (+ 7514us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:13.886756 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:13.887433 (+ 677us) server_negotiation.cc:407] Connection header received
0924 16:17:13.887649 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:13.887659 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:13.887986 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:13.888235 (+ 249us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:17:13.889452 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:13.891567 (+ 2115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:13.896436 (+ 4869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:13.897960 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:13.898115 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:13.899807 (+ 1692us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:13.899876 (+ 69us) server_negotiation.cc:299] Negotiation successful
0924 16:17:13.900029 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7281,"thread_start_us":7129,"threads_started":1}
I20260924 16:17:13.900104 16897 tablet_service.cc:1467] Processing CreateTablet for tablet 7ab61e1f6454441f9880936be0e7d038 (DEFAULT_TABLE table=test-table [id=ab721f48c93f41c49764641265947357]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:17:13.901710 16897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ab61e1f6454441f9880936be0e7d038. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:17:13.902778 17251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:13.881442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.3:46247 (local address 127.0.0.1:53990)
0924 16:17:13.894817 (+ 13375us) negotiation.cc:107] Waiting for socket to connect
0924 16:17:13.894842 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 16:17:13.895084 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:17:13.896210 (+ 1126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:17:13.896215 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:17:13.896245 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:17:13.896582 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:13.896593 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:13.898453 (+ 1860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:13.898460 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:13.900258 (+ 1798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:13.900270 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:13.901986 (+ 1716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:13.901992 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:13.902155 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:13.902193 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 16:17:13.902305 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 16:17:13.902438 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13240,"thread_start_us":12975,"threads_started":1}
I20260924 16:17:13.902859 17252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:13.881823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53990 (local address 127.29.234.3:46247)
0924 16:17:13.894241 (+ 12418us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:13.894250 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:13.895050 (+ 800us) server_negotiation.cc:407] Connection header received
0924 16:17:13.895613 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:13.895624 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:13.895888 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:13.896947 (+ 1059us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:17:13.897163 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:13.898287 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:13.900469 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:13.901791 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:13.901929 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:13.902142 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:13.902345 (+ 203us) server_negotiation.cc:299] Negotiation successful
0924 16:17:13.902496 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12018,"thread_start_us":12059,"threads_started":1}
I20260924 16:17:13.903963 17033 tablet_service.cc:1467] Processing CreateTablet for tablet 7ab61e1f6454441f9880936be0e7d038 (DEFAULT_TABLE table=test-table [id=ab721f48c93f41c49764641265947357]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:17:13.905385 17033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ab61e1f6454441f9880936be0e7d038. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:17:13.906243 17168 tablet_service.cc:1467] Processing CreateTablet for tablet 7ab61e1f6454441f9880936be0e7d038 (DEFAULT_TABLE table=test-table [id=ab721f48c93f41c49764641265947357]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:17:13.907292 17168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ab61e1f6454441f9880936be0e7d038. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:17:13.916662 17253 tablet_bootstrap.cc:492] T 7ab61e1f6454441f9880936be0e7d038 P d1e8d44dcc5f473aa2ab08cffc94a2eb: Bootstrap starting.
I20260924 16:17:13.918313 17254 tablet_bootstrap.cc:492] T 7ab61e1f6454441f9880936be0e7d038 P 8ec37953e2c24727a41e532cdad10e62: Bootstrap starting.
I20260924 16:17:13.920912 17253 tablet_bootstrap.cc:654] T 7ab61e1f6454441f9880936be0e7d038 P d1e8d44dcc5f473aa2ab08cffc94a2eb: Neither blocks nor log segments found. Creating new log.
I20260924 16:17:13.922042 17255 tablet_bootstrap.cc:492] T 7ab61e1f6454441f9880936be0e7d038 P 99343d3fd9f047d392bc7fdbb500a059: Bootstrap starting.
I20260924 16:17:13.922114 17254 tablet_bootstrap.cc:654] T 7ab61e1f6454441f9880936be0e7d038 P 8ec37953e2c24727a41e532cdad10e62: Neither blocks nor log segments found. Creating new log.
I20260924 16:17:13.922263 17253 log.cc:824] T 7ab61e1f6454441f9880936be0e7d038 P d1e8d44dcc5f473aa2ab08cffc94a2eb: Log is configured to *not* fsync() on all Append() calls
I20260924 16:17:13.923338 17254 log.cc:824] T 7ab61e1f6454441f9880936be0e7d038 P 8ec37953e2c24727a41e532cdad10e62: Log is configured to *not* fsync() on all Append() calls
I20260924 16:17:13.925369 17253 tablet_bootstrap.cc:492] T 7ab61e1f6454441f9880936be0e7d038 P d1e8d44dcc5f473aa2ab08cffc94a2eb: No bootstrap required, opened a new log
I20260924 16:17:13.925446 17255 tablet_bootstrap.cc:654] T 7ab61e1f6454441f9880936be0e7d038 P 99343d3fd9f047d392bc7fdbb500a059: Neither blocks nor log segments found. Creating new log.
I20260924 16:17:13.925642 17253 ts_tablet_manager.cc:1397] T 7ab61e1f6454441f9880936be0e7d038 P d1e8d44dcc5f473aa2ab08cffc94a2eb: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.008s
I20260924 16:17:13.925927 17254 tablet_bootstrap.cc:492] T 7ab61e1f6454441f9880936be0e7d038 P 8ec37953e2c24727a41e532cdad10e62: No bootstrap required, opened a new log
I20260924 16:17:13.926194 17254 ts_tablet_manager.cc:1397] T 7ab61e1f6454441f9880936be0e7d038 P 8ec37953e2c24727a41e532cdad10e62: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260924 16:17:13.927858 17255 log.cc:824] T 7ab61e1f6454441f9880936be0e7d038 P 99343d3fd9f047d392bc7fdbb500a059: Log is configured to *not* fsync() on all Append() calls
I20260924 16:17:13.930872 17255 tablet_bootstrap.cc:492] T 7ab61e1f6454441f9880936be0e7d038 P 99343d3fd9f047d392bc7fdbb500a059: No bootstrap required, opened a new log
I20260924 16:17:13.931116 17255 ts_tablet_manager.cc:1397] T 7ab61e1f6454441f9880936be0e7d038 P 99343d3fd9f047d392bc7fdbb500a059: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.005s
I20260924 16:17:13.933609 17254 raft_consensus.cc:348] T 7ab61e1f6454441f9880936be0e7d038 P 8ec37953e2c24727a41e532cdad10e62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99343d3fd9f047d392bc7fdbb500a059" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36293 } } peers { permanent_uuid: "d1e8d44dcc5f473aa2ab08cffc94a2eb" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 33341 } } peers { permanent_uuid: "8ec37953e2c24727a41e532cdad10e62" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 46247 } }
I20260924 16:17:13.933957 17254 raft_consensus.cc:374] T 7ab61e1f6454441f9880936be0e7d038 P 8ec37953e2c24727a41e532cdad10e62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:17:13.934075 17254 raft_consensus.cc:729] T 7ab61e1f6454441f9880936be0e7d038 P 8ec37953e2c24727a41e532cdad10e62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ec37953e2c24727a41e532cdad10e62, State: Initialized, Role: FOLLOWER
I20260924 16:17:13.934703 17254 consensus_queue.cc:260] T 7ab61e1f6454441f9880936be0e7d038 P 8ec37953e2c24727a41e532cdad10e62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99343d3fd9f047d392bc7fdbb500a059" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36293 } } peers { permanent_uuid: "d1e8d44dcc5f473aa2ab08cffc94a2eb" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 33341 } } peers { permanent_uuid: "8ec37953e2c24727a41e532cdad10e62" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 46247 } }
I20260924 16:17:13.935864 17234 heartbeater.cc:503] Master 127.29.234.62:33227 was elected leader, sending a full tablet report...
I20260924 16:17:13.936353 17254 ts_tablet_manager.cc:1428] T 7ab61e1f6454441f9880936be0e7d038 P 8ec37953e2c24727a41e532cdad10e62: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260924 16:17:13.936851 17253 raft_consensus.cc:348] T 7ab61e1f6454441f9880936be0e7d038 P d1e8d44dcc5f473aa2ab08cffc94a2eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99343d3fd9f047d392bc7fdbb500a059" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36293 } } peers { permanent_uuid: "d1e8d44dcc5f473aa2ab08cffc94a2eb" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 33341 } } peers { permanent_uuid: "8ec37953e2c24727a41e532cdad10e62" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 46247 } }
I20260924 16:17:13.937196 17253 raft_consensus.cc:374] T 7ab61e1f6454441f9880936be0e7d038 P d1e8d44dcc5f473aa2ab08cffc94a2eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:17:13.937296 17253 raft_consensus.cc:729] T 7ab61e1f6454441f9880936be0e7d038 P d1e8d44dcc5f473aa2ab08cffc94a2eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e8d44dcc5f473aa2ab08cffc94a2eb, State: Initialized, Role: FOLLOWER
I20260924 16:17:13.937918 17253 consensus_queue.cc:260] T 7ab61e1f6454441f9880936be0e7d038 P d1e8d44dcc5f473aa2ab08cffc94a2eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99343d3fd9f047d392bc7fdbb500a059" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36293 } } peers { permanent_uuid: "d1e8d44dcc5f473aa2ab08cffc94a2eb" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 33341 } } peers { permanent_uuid: "8ec37953e2c24727a41e532cdad10e62" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 46247 } }
I20260924 16:17:13.939347 17253 ts_tablet_manager.cc:1428] T 7ab61e1f6454441f9880936be0e7d038 P d1e8d44dcc5f473aa2ab08cffc94a2eb: Time spent starting tablet: real 0.014s user 0.007s sys 0.006s
I20260924 16:17:13.943130 17255 raft_consensus.cc:348] T 7ab61e1f6454441f9880936be0e7d038 P 99343d3fd9f047d392bc7fdbb500a059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99343d3fd9f047d392bc7fdbb500a059" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36293 } } peers { permanent_uuid: "d1e8d44dcc5f473aa2ab08cffc94a2eb" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 33341 } } peers { permanent_uuid: "8ec37953e2c24727a41e532cdad10e62" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 46247 } }
I20260924 16:17:13.943480 17255 raft_consensus.cc:374] T 7ab61e1f6454441f9880936be0e7d038 P 99343d3fd9f047d392bc7fdbb500a059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:17:13.943614 17255 raft_consensus.cc:729] T 7ab61e1f6454441f9880936be0e7d038 P 99343d3fd9f047d392bc7fdbb500a059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99343d3fd9f047d392bc7fdbb500a059, State: Initialized, Role: FOLLOWER
I20260924 16:17:13.944123 17255 consensus_queue.cc:260] T 7ab61e1f6454441f9880936be0e7d038 P 99343d3fd9f047d392bc7fdbb500a059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99343d3fd9f047d392bc7fdbb500a059" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36293 } } peers { permanent_uuid: "d1e8d44dcc5f473aa2ab08cffc94a2eb" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 33341 } } peers { permanent_uuid: "8ec37953e2c24727a41e532cdad10e62" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 46247 } }
I20260924 16:17:13.945021 17099 heartbeater.cc:503] Master 127.29.234.62:33227 was elected leader, sending a full tablet report...
I20260924 16:17:13.945590 17255 ts_tablet_manager.cc:1428] T 7ab61e1f6454441f9880936be0e7d038 P 99343d3fd9f047d392bc7fdbb500a059: Time spent starting tablet: real 0.014s user 0.007s sys 0.008s
I20260924 16:17:13.986274 17260 raft_consensus.cc:482] T 7ab61e1f6454441f9880936be0e7d038 P d1e8d44dcc5f473aa2ab08cffc94a2eb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:17:13.986601 17260 raft_consensus.cc:504] T 7ab61e1f6454441f9880936be0e7d038 P d1e8d44dcc5f473aa2ab08cffc94a2eb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99343d3fd9f047d392bc7fdbb500a059" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36293 } } peers { permanent_uuid: "d1e8d44dcc5f473aa2ab08cffc94a2eb" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 33341 } } peers { permanent_uuid: "8ec37953e2c24727a41e532cdad10e62" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 46247 } }
I20260924 16:17:13.988339 17260 leader_election.cc:288] T 7ab61e1f6454441f9880936be0e7d038 P d1e8d44dcc5f473aa2ab08cffc94a2eb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99343d3fd9f047d392bc7fdbb500a059 (127.29.234.2:36293), 8ec37953e2c24727a41e532cdad10e62 (127.29.234.3:46247)
I20260924 16:17:13.997745 17252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:13.988869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.1:48667 (local address 127.29.234.3:46247)
0924 16:17:13.989049 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:13.989054 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:13.989779 (+ 725us) server_negotiation.cc:407] Connection header received
0924 16:17:13.989998 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:13.990002 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:13.990072 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:13.990172 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:17:13.991107 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:13.992233 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:13.995346 (+ 3113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:13.996556 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:13.996723 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:13.996786 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:13.997332 (+ 546us) server_negotiation.cc:299] Negotiation successful
0924 16:17:13.997483 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 16:17:13.997745 17248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:13.988748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.1:57317 (local address 127.29.234.2:36293)
0924 16:17:13.988948 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:13.988954 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:13.989828 (+ 874us) server_negotiation.cc:407] Connection header received
0924 16:17:13.989989 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:13.989995 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:13.990072 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:13.990188 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:17:13.991076 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:13.992225 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:13.995335 (+ 3110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:13.996558 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:13.996713 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:13.996784 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:13.997332 (+ 548us) server_negotiation.cc:299] Negotiation successful
0924 16:17:13.997483 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 16:17:13.997994 17262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:13.988721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.2:36293 (local address 127.29.234.1:57317)
0924 16:17:13.989654 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0924 16:17:13.989675 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 16:17:13.989826 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:17:13.990206 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:17:13.990211 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:17:13.990251 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:17:13.990855 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:13.990877 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:13.992375 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:13.992381 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:13.995168 (+ 2787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:13.995178 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:13.996805 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:13.996817 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:13.997076 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:13.997182 (+ 106us) client_negotiation.cc:769] Sending connection context
0924 16:17:13.997281 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 16:17:13.997497 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":839,"mutex_wait_us":365,"spinlock_wait_cycles":5120,"thread_start_us":527,"threads_started":1}
I20260924 16:17:13.998310 17263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:13.988570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.3:46247 (local address 127.29.234.1:48667)
0924 16:17:13.989654 (+ 1084us) negotiation.cc:107] Waiting for socket to connect
0924 16:17:13.989675 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 16:17:13.989869 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:17:13.990206 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:17:13.990211 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:17:13.990245 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:17:13.990964 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:13.990978 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:13.992375 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:13.992381 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:13.995148 (+ 2767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:13.995169 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:13.996832 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:13.996837 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:13.997075 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:13.997188 (+ 113us) client_negotiation.cc:769] Sending connection context
0924 16:17:13.997281 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 16:17:13.997497 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":941,"mutex_wait_us":382,"spinlock_wait_cycles":41344,"thread_start_us":712,"threads_started":1}
I20260924 16:17:13.999046 17053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ab61e1f6454441f9880936be0e7d038" candidate_uuid: "d1e8d44dcc5f473aa2ab08cffc94a2eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99343d3fd9f047d392bc7fdbb500a059" is_pre_election: true
I20260924 16:17:13.999301 17188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ab61e1f6454441f9880936be0e7d038" candidate_uuid: "d1e8d44dcc5f473aa2ab08cffc94a2eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ec37953e2c24727a41e532cdad10e62" is_pre_election: true
I20260924 16:17:13.999548 17053 raft_consensus.cc:2454] T 7ab61e1f6454441f9880936be0e7d038 P 99343d3fd9f047d392bc7fdbb500a059 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e8d44dcc5f473aa2ab08cffc94a2eb in term 0.
I20260924 16:17:13.999822 17188 raft_consensus.cc:2454] T 7ab61e1f6454441f9880936be0e7d038 P 8ec37953e2c24727a41e532cdad10e62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e8d44dcc5f473aa2ab08cffc94a2eb in term 0.
I20260924 16:17:14.000423 16853 leader_election.cc:302] T 7ab61e1f6454441f9880936be0e7d038 P d1e8d44dcc5f473aa2ab08cffc94a2eb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99343d3fd9f047d392bc7fdbb500a059, d1e8d44dcc5f473aa2ab08cffc94a2eb; no voters:
W20260924 16:17:14.000653 16964 tablet.cc:2369] T 7ab61e1f6454441f9880936be0e7d038 P d1e8d44dcc5f473aa2ab08cffc94a2eb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:17:14.000921 17260 raft_consensus.cc:2781] T 7ab61e1f6454441f9880936be0e7d038 P d1e8d44dcc5f473aa2ab08cffc94a2eb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:17:14.001107 17260 raft_consensus.cc:482] T 7ab61e1f6454441f9880936be0e7d038 P d1e8d44dcc5f473aa2ab08cffc94a2eb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:17:14.001209 17260 raft_consensus.cc:3037] T 7ab61e1f6454441f9880936be0e7d038 P d1e8d44dcc5f473aa2ab08cffc94a2eb [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:17:14.003374 17260 raft_consensus.cc:504] T 7ab61e1f6454441f9880936be0e7d038 P d1e8d44dcc5f473aa2ab08cffc94a2eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99343d3fd9f047d392bc7fdbb500a059" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36293 } } peers { permanent_uuid: "d1e8d44dcc5f473aa2ab08cffc94a2eb" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 33341 } } peers { permanent_uuid: "8ec37953e2c24727a41e532cdad10e62" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 46247 } }
I20260924 16:17:14.003935 17260 leader_election.cc:288] T 7ab61e1f6454441f9880936be0e7d038 P d1e8d44dcc5f473aa2ab08cffc94a2eb [CANDIDATE]: Term 1 election: Requested vote from peers 99343d3fd9f047d392bc7fdbb500a059 (127.29.234.2:36293), 8ec37953e2c24727a41e532cdad10e62 (127.29.234.3:46247)
I20260924 16:17:14.004359 17053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ab61e1f6454441f9880936be0e7d038" candidate_uuid: "d1e8d44dcc5f473aa2ab08cffc94a2eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99343d3fd9f047d392bc7fdbb500a059"
I20260924 16:17:14.004464 17188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ab61e1f6454441f9880936be0e7d038" candidate_uuid: "d1e8d44dcc5f473aa2ab08cffc94a2eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ec37953e2c24727a41e532cdad10e62"
I20260924 16:17:14.004566 17053 raft_consensus.cc:3037] T 7ab61e1f6454441f9880936be0e7d038 P 99343d3fd9f047d392bc7fdbb500a059 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:17:14.004714 17188 raft_consensus.cc:3037] T 7ab61e1f6454441f9880936be0e7d038 P 8ec37953e2c24727a41e532cdad10e62 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:17:14.006899 17053 raft_consensus.cc:2454] T 7ab61e1f6454441f9880936be0e7d038 P 99343d3fd9f047d392bc7fdbb500a059 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e8d44dcc5f473aa2ab08cffc94a2eb in term 1.
I20260924 16:17:14.006951 17188 raft_consensus.cc:2454] T 7ab61e1f6454441f9880936be0e7d038 P 8ec37953e2c24727a41e532cdad10e62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e8d44dcc5f473aa2ab08cffc94a2eb in term 1.
I20260924 16:17:14.007311 16853 leader_election.cc:302] T 7ab61e1f6454441f9880936be0e7d038 P d1e8d44dcc5f473aa2ab08cffc94a2eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99343d3fd9f047d392bc7fdbb500a059, d1e8d44dcc5f473aa2ab08cffc94a2eb; no voters:
I20260924 16:17:14.007632 17260 raft_consensus.cc:2781] T 7ab61e1f6454441f9880936be0e7d038 P d1e8d44dcc5f473aa2ab08cffc94a2eb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:17:14.008091 17260 raft_consensus.cc:686] T 7ab61e1f6454441f9880936be0e7d038 P d1e8d44dcc5f473aa2ab08cffc94a2eb [term 1 LEADER]: Becoming Leader. State: Replica: d1e8d44dcc5f473aa2ab08cffc94a2eb, State: Running, Role: LEADER
I20260924 16:17:14.008633 17260 consensus_queue.cc:237] T 7ab61e1f6454441f9880936be0e7d038 P d1e8d44dcc5f473aa2ab08cffc94a2eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99343d3fd9f047d392bc7fdbb500a059" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36293 } } peers { permanent_uuid: "d1e8d44dcc5f473aa2ab08cffc94a2eb" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 33341 } } peers { permanent_uuid: "8ec37953e2c24727a41e532cdad10e62" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 46247 } }
I20260924 16:17:14.013177 16770 catalog_manager.cc:5495] T 7ab61e1f6454441f9880936be0e7d038 P d1e8d44dcc5f473aa2ab08cffc94a2eb reported cstate change: term changed from 0 to 1, leader changed from <none> to d1e8d44dcc5f473aa2ab08cffc94a2eb (127.29.234.1). New cstate: current_term: 1 leader_uuid: "d1e8d44dcc5f473aa2ab08cffc94a2eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99343d3fd9f047d392bc7fdbb500a059" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e8d44dcc5f473aa2ab08cffc94a2eb" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 33341 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ec37953e2c24727a41e532cdad10e62" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 46247 } health_report { overall_health: UNKNOWN } } }
W20260924 16:17:14.017681 17100 tablet.cc:2369] T 7ab61e1f6454441f9880936be0e7d038 P 99343d3fd9f047d392bc7fdbb500a059: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:17:14.030274 17250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:14.025165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52986 (local address 127.29.234.1:33341)
0924 16:17:14.025383 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:14.025392 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:14.025510 (+ 118us) server_negotiation.cc:407] Connection header received
0924 16:17:14.025670 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:14.025674 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:14.025742 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:14.025862 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:17:14.026233 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:14.026921 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:14.028235 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:14.028665 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:14.028780 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:14.029227 (+ 447us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:17:14.029783 (+ 556us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:17:14.029917 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:14.030051 (+ 134us) server_negotiation.cc:299] Negotiation successful
0924 16:17:14.030089 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 16:17:14.038144 17188 raft_consensus.cc:1261] T 7ab61e1f6454441f9880936be0e7d038 P 8ec37953e2c24727a41e532cdad10e62 [term 1 FOLLOWER]: Refusing update from remote peer d1e8d44dcc5f473aa2ab08cffc94a2eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:17:14.038183 17053 raft_consensus.cc:1261] T 7ab61e1f6454441f9880936be0e7d038 P 99343d3fd9f047d392bc7fdbb500a059 [term 1 FOLLOWER]: Refusing update from remote peer d1e8d44dcc5f473aa2ab08cffc94a2eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:17:14.039143 17265 consensus_queue.cc:1035] T 7ab61e1f6454441f9880936be0e7d038 P d1e8d44dcc5f473aa2ab08cffc94a2eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "99343d3fd9f047d392bc7fdbb500a059" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:17:14.039479 17260 consensus_queue.cc:1035] T 7ab61e1f6454441f9880936be0e7d038 P d1e8d44dcc5f473aa2ab08cffc94a2eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ec37953e2c24727a41e532cdad10e62" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 46247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:17:14.050745 17268 mvcc.cc:205] Tried to move back new op lower bound from 7332932133010575360 to 7332932132903317504. Current Snapshot: MvccSnapshot[applied={T|T < 7332932133010575360}]
I20260924 16:17:14.053485 17269 mvcc.cc:205] Tried to move back new op lower bound from 7332932133010575360 to 7332932132903317504. Current Snapshot: MvccSnapshot[applied={T|T < 7332932133010575360}]
I20260924 16:17:14.067649 16770 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:17:14.067940 16770 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:17:14.071400 16770 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 34b6818dc7ab4bc2a81644561a3c67f3: Sending DeleteTablet for 3 replicas of tablet 7ab61e1f6454441f9880936be0e7d038
I20260924 16:17:14.072549 16897 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ab61e1f6454441f9880936be0e7d038 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:17:14 UTC) from {username='kudu', principal='kudu/127.29.234.62@KRBTEST.COM'} at 127.0.0.1:52970
I20260924 16:17:14.072543 17033 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ab61e1f6454441f9880936be0e7d038 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:17:14 UTC) from {username='kudu', principal='kudu/127.29.234.62@KRBTEST.COM'} at 127.0.0.1:37880
I20260924 16:17:14.072868 17168 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ab61e1f6454441f9880936be0e7d038 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:17:14 UTC) from {username='kudu', principal='kudu/127.29.234.62@KRBTEST.COM'} at 127.0.0.1:53990
I20260924 16:17:14.077310 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 16833
I20260924 16:17:14.083824 17279 tablet_replica.cc:330] T 7ab61e1f6454441f9880936be0e7d038 P d1e8d44dcc5f473aa2ab08cffc94a2eb: stopping tablet replica
I20260924 16:17:14.084491 17279 raft_consensus.cc:2229] T 7ab61e1f6454441f9880936be0e7d038 P d1e8d44dcc5f473aa2ab08cffc94a2eb [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:17:14.085372 17279 raft_consensus.cc:2258] T 7ab61e1f6454441f9880936be0e7d038 P d1e8d44dcc5f473aa2ab08cffc94a2eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:17:14.086305 17278 tablet_replica.cc:330] T 7ab61e1f6454441f9880936be0e7d038 P 99343d3fd9f047d392bc7fdbb500a059: stopping tablet replica
I20260924 16:17:14.086815 17278 raft_consensus.cc:2229] T 7ab61e1f6454441f9880936be0e7d038 P 99343d3fd9f047d392bc7fdbb500a059 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:17:14.087368 17278 raft_consensus.cc:2258] T 7ab61e1f6454441f9880936be0e7d038 P 99343d3fd9f047d392bc7fdbb500a059 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:17:14.089905 17279 ts_tablet_manager.cc:1905] T 7ab61e1f6454441f9880936be0e7d038 P d1e8d44dcc5f473aa2ab08cffc94a2eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:17:14.091184 17278 ts_tablet_manager.cc:1905] T 7ab61e1f6454441f9880936be0e7d038 P 99343d3fd9f047d392bc7fdbb500a059: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:17:14.098809 17279 ts_tablet_manager.cc:1918] T 7ab61e1f6454441f9880936be0e7d038 P d1e8d44dcc5f473aa2ab08cffc94a2eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:17:14.098981 17279 log.cc:1196] T 7ab61e1f6454441f9880936be0e7d038 P d1e8d44dcc5f473aa2ab08cffc94a2eb: Deleting WAL directory at /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/minicluster-data/ts-0/wal/wals/7ab61e1f6454441f9880936be0e7d038
I20260924 16:17:14.099099 17278 ts_tablet_manager.cc:1918] T 7ab61e1f6454441f9880936be0e7d038 P 99343d3fd9f047d392bc7fdbb500a059: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:17:14.099334 17278 log.cc:1196] T 7ab61e1f6454441f9880936be0e7d038 P 99343d3fd9f047d392bc7fdbb500a059: Deleting WAL directory at /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/minicluster-data/ts-1/wal/wals/7ab61e1f6454441f9880936be0e7d038
I20260924 16:17:14.099498 17279 ts_tablet_manager.cc:1939] T 7ab61e1f6454441f9880936be0e7d038 P d1e8d44dcc5f473aa2ab08cffc94a2eb: Deleting consensus metadata
I20260924 16:17:14.100046 17278 ts_tablet_manager.cc:1939] T 7ab61e1f6454441f9880936be0e7d038 P 99343d3fd9f047d392bc7fdbb500a059: Deleting consensus metadata
I20260924 16:17:14.101315 17280 tablet_replica.cc:330] T 7ab61e1f6454441f9880936be0e7d038 P 8ec37953e2c24727a41e532cdad10e62: stopping tablet replica
I20260924 16:17:14.101594 16756 catalog_manager.cc:4841] TS d1e8d44dcc5f473aa2ab08cffc94a2eb (127.29.234.1:33341): tablet 7ab61e1f6454441f9880936be0e7d038 (table test-table [id=ab721f48c93f41c49764641265947357]) successfully deleted
I20260924 16:17:14.102247 17280 raft_consensus.cc:2229] T 7ab61e1f6454441f9880936be0e7d038 P 8ec37953e2c24727a41e532cdad10e62 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:17:14.102774 17280 raft_consensus.cc:2258] T 7ab61e1f6454441f9880936be0e7d038 P 8ec37953e2c24727a41e532cdad10e62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:17:14.103451 16757 catalog_manager.cc:4841] TS 99343d3fd9f047d392bc7fdbb500a059 (127.29.234.2:36293): tablet 7ab61e1f6454441f9880936be0e7d038 (table test-table [id=ab721f48c93f41c49764641265947357]) successfully deleted
I20260924 16:17:14.105093 17280 ts_tablet_manager.cc:1905] T 7ab61e1f6454441f9880936be0e7d038 P 8ec37953e2c24727a41e532cdad10e62: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 24 16:17:14 dist-test-slave-v5j9 krb5kdc[16725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.234.1@KRBTEST.COM
I20260924 16:17:14.112152 17280 ts_tablet_manager.cc:1918] T 7ab61e1f6454441f9880936be0e7d038 P 8ec37953e2c24727a41e532cdad10e62: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:17:14.112314 17280 log.cc:1196] T 7ab61e1f6454441f9880936be0e7d038 P 8ec37953e2c24727a41e532cdad10e62: Deleting WAL directory at /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790266323169505-30632-0/minicluster-data/ts-2/wal/wals/7ab61e1f6454441f9880936be0e7d038
I20260924 16:17:14.112876 17280 ts_tablet_manager.cc:1939] T 7ab61e1f6454441f9880936be0e7d038 P 8ec37953e2c24727a41e532cdad10e62: Deleting consensus metadata
I20260924 16:17:14.112963 17250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:14.080573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52988 (local address 127.29.234.1:33341)
0924 16:17:14.080787 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:14.080793 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:14.095046 (+ 14253us) server_negotiation.cc:407] Connection header received
0924 16:17:14.095286 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:14.095303 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:14.095388 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:14.095491 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:14.097248 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:14.098488 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:14.101113 (+ 2625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:14.101851 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:14.101959 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:14.106767 (+ 4808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:14.106778 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:14.106802 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:17:14.106833 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:14.109989 (+ 3156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:17:14.110886 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:17:14.110894 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:17:14.110898 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:17:14.111047 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:17:14.111438 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:17:14.111443 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:17:14.111446 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:17:14.111814 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:14.112106 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:14.112491 (+ 385us) server_negotiation.cc:299] Negotiation successful
0924 16:17:14.112675 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 16:17:14.114379 16958 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:17:14.114789 16755 catalog_manager.cc:4841] TS 8ec37953e2c24727a41e532cdad10e62 (127.29.234.3:46247): tablet 7ab61e1f6454441f9880936be0e7d038 (table test-table [id=ab721f48c93f41c49764641265947357]) successfully deleted
I20260924 16:17:14.213706 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 16833
I20260924 16:17:14.232977 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 16969
Sep 24 16:17:14 dist-test-slave-v5j9 krb5kdc[16725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.234.2@KRBTEST.COM
I20260924 16:17:14.248131 17248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:14.233797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37886 (local address 127.29.234.2:36293)
0924 16:17:14.233959 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:14.233964 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:14.234066 (+ 102us) server_negotiation.cc:407] Connection header received
0924 16:17:14.234287 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:14.234293 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:14.234366 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:14.234449 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:14.235723 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:14.236411 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:14.237967 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:14.238341 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:14.238433 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:14.242759 (+ 4326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:14.242773 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:14.242802 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:17:14.242856 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:14.245761 (+ 2905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:17:14.246412 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:17:14.246420 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:17:14.246422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:17:14.246537 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:17:14.246900 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:17:14.246904 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:17:14.246905 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:17:14.247260 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:14.247475 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:14.247725 (+ 250us) server_negotiation.cc:299] Negotiation successful
0924 16:17:14.247885 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 16:17:14.248775 17094 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:17:14.338631 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 16969
I20260924 16:17:14.356660 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 17104
Sep 24 16:17:14 dist-test-slave-v5j9 krb5kdc[16725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.234.3@KRBTEST.COM
I20260924 16:17:14.371668 17252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:14.357462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54006 (local address 127.29.234.3:46247)
0924 16:17:14.357658 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:14.357664 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:14.357684 (+ 20us) server_negotiation.cc:407] Connection header received
0924 16:17:14.357889 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:14.357893 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:14.357957 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:14.358063 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:14.359396 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:14.360144 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:14.361656 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:14.362041 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:14.362141 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:14.366298 (+ 4157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:14.366314 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:14.366332 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:17:14.366364 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:14.369143 (+ 2779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:17:14.369884 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:17:14.369891 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:17:14.369894 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:17:14.370015 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:17:14.370341 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:17:14.370344 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:17:14.370346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:17:14.370667 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:14.370879 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:14.371210 (+ 331us) server_negotiation.cc:299] Negotiation successful
0924 16:17:14.371424 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 16:17:14.372325 17229 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:17:14.464974 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 17104
I20260924 16:17:14.482599 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 16738
I20260924 16:17:14.483752 16802 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:17:14.597198 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 16738
2026-09-24T16:17:14Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4302 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-24T16:17:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:17:14Z Disabled control of system clock
I20260924 16:17:14.675354 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266323169505-30632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266323169505-30632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266323169505-30632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:43915
--webserver_interface=127.29.234.62
--webserver_port=0
--builtin_ntp_servers=127.29.234.20:38859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.234.62:43915
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 16:17:15.011426 17289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:17:15.011704 17289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:17:15.011864 17289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:17:15.019260 17289 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:17:15.019371 17289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:17:15.019445 17289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:17:15.019532 17289 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:17:15.019613 17289 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:17:15.029204 17289 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:38859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266323169505-30632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266323169505-30632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.234.62:43915
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:43915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.234.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266323169505-30632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:17:15.030699 17289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:17:15.032622 17289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:17:15.045418 17294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:15.045495 17297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:15.045471 17295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:17:15.046567 17289 server_base.cc:1034] running on GCE node
I20260924 16:17:15.047459 17289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:17:15.049479 17289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:17:15.050729 17289 hybrid_clock.cc:648] HybridClock initialized: now 1790266635050675 us; error 59 us; skew 500 ppm
I20260924 16:17:15.051126 17289 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 16:17:15.053571 17289 webserver.cc:460] Webserver started at http://127.29.234.62:40165/ using document root <none> and password file <none>
I20260924 16:17:15.054373 17289 fs_manager.cc:359] Metadata directory not provided
I20260924 16:17:15.054514 17289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:17:15.054860 17289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:17:15.058024 17289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266323169505-30632-0/minicluster-data/master-0/data/instance:
uuid: "204b69aeb9eb425e93eb446e45ef6342"
format_stamp: "Formatted at 2026-09-24 16:17:15 on dist-test-slave-v5j9"
server_key: "b9a65636216de8f06b5683f91632105e"
server_key_iv: "190919dd0260249bc81e4a3c22c2a0ff"
server_key_version: "encryptionkey@0"
I20260924 16:17:15.058881 17289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266323169505-30632-0/minicluster-data/master-0/wal/instance:
uuid: "204b69aeb9eb425e93eb446e45ef6342"
format_stamp: "Formatted at 2026-09-24 16:17:15 on dist-test-slave-v5j9"
server_key: "b9a65636216de8f06b5683f91632105e"
server_key_iv: "190919dd0260249bc81e4a3c22c2a0ff"
server_key_version: "encryptionkey@0"
I20260924 16:17:15.064246 17289 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 16:17:15.067893 17303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:17:15.069442 17289 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 16:17:15.069592 17289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266323169505-30632-0/minicluster-data/master-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266323169505-30632-0/minicluster-data/master-0/wal
uuid: "204b69aeb9eb425e93eb446e45ef6342"
format_stamp: "Formatted at 2026-09-24 16:17:15 on dist-test-slave-v5j9"
server_key: "b9a65636216de8f06b5683f91632105e"
server_key_iv: "190919dd0260249bc81e4a3c22c2a0ff"
server_key_version: "encryptionkey@0"
I20260924 16:17:15.069777 17289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266323169505-30632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266323169505-30632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266323169505-30632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:17:15.096328 17289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:17:15.097222 17289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:17:15.097496 17289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:17:15.116132 17289 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.62:43915
I20260924 16:17:15.116153 17355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.62:43915 every 8 connection(s)
I20260924 16:17:15.117579 17289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
I20260924 16:17:15.119165 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 17289
I20260924 16:17:15.119342 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266323169505-30632-0/minicluster-data/master-0/wal/instance
I20260924 16:17:15.119818 30632 external_mini_cluster.cc:1428] Setting key 938c7c1c0b47c2da417ca9d33c183a74
I20260924 16:17:15.122098 17356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:17:15.135756 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:15.121853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56044 (local address 127.29.234.62:43915)
0924 16:17:15.122953 (+ 1100us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:15.122967 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:15.123019 (+ 52us) server_negotiation.cc:407] Connection header received
0924 16:17:15.125174 (+ 2155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:15.125202 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:15.125866 (+ 664us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:15.126309 (+ 443us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:15.126865 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:15.128927 (+ 2062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:15.130351 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:15.131366 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:15.131554 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:15.132731 (+ 1177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:15.132741 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:15.132761 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:17:15.133581 (+ 820us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:15.133679 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:17:15.133710 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:15.133857 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:15.134071 (+ 214us) server_negotiation.cc:299] Negotiation successful
0924 16:17:15.134446 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":341,"threads_started":1}
I20260924 16:17:15.136826 17356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 204b69aeb9eb425e93eb446e45ef6342: Bootstrap starting.
I20260924 16:17:15.139997 17356 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 204b69aeb9eb425e93eb446e45ef6342: Neither blocks nor log segments found. Creating new log.
I20260924 16:17:15.141019 17356 log.cc:824] T 00000000000000000000000000000000 P 204b69aeb9eb425e93eb446e45ef6342: Log is configured to *not* fsync() on all Append() calls
I20260924 16:17:15.144042 17356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 204b69aeb9eb425e93eb446e45ef6342: No bootstrap required, opened a new log
I20260924 16:17:15.151535 17356 raft_consensus.cc:348] T 00000000000000000000000000000000 P 204b69aeb9eb425e93eb446e45ef6342 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "204b69aeb9eb425e93eb446e45ef6342" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 43915 } }
I20260924 16:17:15.151957 17356 raft_consensus.cc:374] T 00000000000000000000000000000000 P 204b69aeb9eb425e93eb446e45ef6342 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:17:15.152070 17356 raft_consensus.cc:729] T 00000000000000000000000000000000 P 204b69aeb9eb425e93eb446e45ef6342 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 204b69aeb9eb425e93eb446e45ef6342, State: Initialized, Role: FOLLOWER
I20260924 16:17:15.152716 17356 consensus_queue.cc:260] T 00000000000000000000000000000000 P 204b69aeb9eb425e93eb446e45ef6342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "204b69aeb9eb425e93eb446e45ef6342" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 43915 } }
I20260924 16:17:15.152969 17356 raft_consensus.cc:388] T 00000000000000000000000000000000 P 204b69aeb9eb425e93eb446e45ef6342 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:17:15.153115 17356 raft_consensus.cc:482] T 00000000000000000000000000000000 P 204b69aeb9eb425e93eb446e45ef6342 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:17:15.153302 17356 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 204b69aeb9eb425e93eb446e45ef6342 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:17:15.155592 17356 raft_consensus.cc:504] T 00000000000000000000000000000000 P 204b69aeb9eb425e93eb446e45ef6342 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "204b69aeb9eb425e93eb446e45ef6342" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 43915 } }
I20260924 16:17:15.156111 17356 leader_election.cc:302] T 00000000000000000000000000000000 P 204b69aeb9eb425e93eb446e45ef6342 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 204b69aeb9eb425e93eb446e45ef6342; no voters:
I20260924 16:17:15.156690 17356 leader_election.cc:288] T 00000000000000000000000000000000 P 204b69aeb9eb425e93eb446e45ef6342 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:17:15.156850 17361 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 204b69aeb9eb425e93eb446e45ef6342 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:17:15.157699 17361 raft_consensus.cc:686] T 00000000000000000000000000000000 P 204b69aeb9eb425e93eb446e45ef6342 [term 1 LEADER]: Becoming Leader. State: Replica: 204b69aeb9eb425e93eb446e45ef6342, State: Running, Role: LEADER
I20260924 16:17:15.158264 17361 consensus_queue.cc:237] T 00000000000000000000000000000000 P 204b69aeb9eb425e93eb446e45ef6342 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "204b69aeb9eb425e93eb446e45ef6342" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 43915 } }
I20260924 16:17:15.158923 17356 sys_catalog.cc:564] T 00000000000000000000000000000000 P 204b69aeb9eb425e93eb446e45ef6342 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:17:15.161124 17362 sys_catalog.cc:455] T 00000000000000000000000000000000 P 204b69aeb9eb425e93eb446e45ef6342 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "204b69aeb9eb425e93eb446e45ef6342" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "204b69aeb9eb425e93eb446e45ef6342" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 43915 } } }
I20260924 16:17:15.161584 17362 sys_catalog.cc:458] T 00000000000000000000000000000000 P 204b69aeb9eb425e93eb446e45ef6342 [sys.catalog]: This master's current role is: LEADER
I20260924 16:17:15.165630 17363 sys_catalog.cc:455] T 00000000000000000000000000000000 P 204b69aeb9eb425e93eb446e45ef6342 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 204b69aeb9eb425e93eb446e45ef6342. Latest consensus state: current_term: 1 leader_uuid: "204b69aeb9eb425e93eb446e45ef6342" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "204b69aeb9eb425e93eb446e45ef6342" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 43915 } } }
I20260924 16:17:15.166157 17363 sys_catalog.cc:458] T 00000000000000000000000000000000 P 204b69aeb9eb425e93eb446e45ef6342 [sys.catalog]: This master's current role is: LEADER
I20260924 16:17:15.166879 17370 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:17:15.172317 17370 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:17:15.181756 17370 catalog_manager.cc:1347] Generated new cluster ID: 3a779f0529364b1182b2353005d6e8d2
I20260924 16:17:15.181915 17370 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:17:15.205457 17370 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:17:15.206722 17370 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:17:15.212576 17370 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 204b69aeb9eb425e93eb446e45ef6342: Generated new TSK 0
I20260924 16:17:15.213459 17370 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:17:15.226428 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266323169505-30632-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266323169505-30632-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266323169505-30632-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.1:0
--local_ip_for_outbound_sockets=127.29.234.1
--webserver_interface=127.29.234.1
--webserver_port=0
--tserver_master_addrs=127.29.234.62:43915
--builtin_ntp_servers=127.29.234.20:38859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 16:17:15.584940 17380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:17:15.585235 17380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:17:15.585321 17380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:17:15.592695 17380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:17:15.592808 17380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:17:15.592923 17380 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.1
I20260924 16:17:15.602479 17380 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:38859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266323169505-30632-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266323169505-30632-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.234.1
--webserver_port=0
--tserver_master_addrs=127.29.234.62:43915
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.1
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266323169505-30632-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:17:15.603871 17380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:17:15.605707 17380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:17:15.619318 17385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:15.619320 17386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:15.619354 17388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:17:15.620290 17380 server_base.cc:1034] running on GCE node
I20260924 16:17:15.620970 17380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:17:15.622059 17380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:17:15.623299 17380 hybrid_clock.cc:648] HybridClock initialized: now 1790266635623248 us; error 60 us; skew 500 ppm
I20260924 16:17:15.623633 17380 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 16:17:15.625995 17380 webserver.cc:460] Webserver started at http://127.29.234.1:42097/ using document root <none> and password file <none>
I20260924 16:17:15.626745 17380 fs_manager.cc:359] Metadata directory not provided
I20260924 16:17:15.626879 17380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:17:15.627274 17380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:17:15.630285 17380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266323169505-30632-0/minicluster-data/ts-0/data/instance:
uuid: "1d315e0deca04ed79705e9671e85f206"
format_stamp: "Formatted at 2026-09-24 16:17:15 on dist-test-slave-v5j9"
server_key: "9efb50ebe3bc25346fb8e95d650d1f43"
server_key_iv: "32da8259f2ed8c917736ab8a75bb96b0"
server_key_version: "encryptionkey@0"
I20260924 16:17:15.631096 17380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266323169505-30632-0/minicluster-data/ts-0/wal/instance:
uuid: "1d315e0deca04ed79705e9671e85f206"
format_stamp: "Formatted at 2026-09-24 16:17:15 on dist-test-slave-v5j9"
server_key: "9efb50ebe3bc25346fb8e95d650d1f43"
server_key_iv: "32da8259f2ed8c917736ab8a75bb96b0"
server_key_version: "encryptionkey@0"
I20260924 16:17:15.636759 17380 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 16:17:15.640458 17394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:17:15.641901 17380 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:17:15.642086 17380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266323169505-30632-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266323169505-30632-0/minicluster-data/ts-0/wal
uuid: "1d315e0deca04ed79705e9671e85f206"
format_stamp: "Formatted at 2026-09-24 16:17:15 on dist-test-slave-v5j9"
server_key: "9efb50ebe3bc25346fb8e95d650d1f43"
server_key_iv: "32da8259f2ed8c917736ab8a75bb96b0"
server_key_version: "encryptionkey@0"
I20260924 16:17:15.642325 17380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266323169505-30632-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266323169505-30632-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266323169505-30632-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:17:15.658324 17380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:17:15.659247 17380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:17:15.659567 17380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:17:15.660619 17380 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:17:15.662453 17380 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:17:15.662575 17380 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:17:15.662712 17380 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:17:15.662802 17380 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:17:15.696542 17380 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.1:40741
I20260924 16:17:15.696599 17508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.1:40741 every 8 connection(s)
I20260924 16:17:15.698011 17380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
I20260924 16:17:15.698673 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 17380
I20260924 16:17:15.698885 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266323169505-30632-0/minicluster-data/ts-0/wal/instance
I20260924 16:17:15.699471 30632 external_mini_cluster.cc:1428] Setting key b4d17ac1c9960f1e4592c3774f273569
I20260924 16:17:15.702920 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266323169505-30632-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266323169505-30632-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266323169505-30632-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.2:0
--local_ip_for_outbound_sockets=127.29.234.2
--webserver_interface=127.29.234.2
--webserver_port=0
--tserver_master_addrs=127.29.234.62:43915
--builtin_ntp_servers=127.29.234.20:38859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 16:17:15.711709 17511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:15.700842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.1:40271 (local address 127.29.234.62:43915)
0924 16:17:15.701320 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:15.701325 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:15.702890 (+ 1565us) server_negotiation.cc:407] Connection header received
0924 16:17:15.704819 (+ 1929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:15.704824 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:15.704919 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:15.705039 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:15.706261 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:15.706992 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:15.708643 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:15.709047 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:15.709233 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:15.710862 (+ 1629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:15.710867 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:15.710871 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:17:15.710890 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:15.710933 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:17:15.710945 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:15.711022 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:15.711286 (+ 264us) server_negotiation.cc:299] Negotiation successful
0924 16:17:15.711366 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":271,"threads_started":1}
I20260924 16:17:15.713145 17512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:15.701708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:43915 (local address 127.29.234.1:40271)
0924 16:17:15.702723 (+ 1015us) negotiation.cc:107] Waiting for socket to connect
0924 16:17:15.702764 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 16:17:15.704537 (+ 1773us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:17:15.705304 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:17:15.705313 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:17:15.705475 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:17:15.706093 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:15.706111 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:15.707151 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:15.707157 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:15.708498 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:15.708511 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:15.709327 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:15.709337 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:15.709672 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:15.710474 (+ 802us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:17:15.710479 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:17:15.710539 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:17:15.710649 (+ 110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:17:15.711052 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:17:15.711065 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:17:15.711076 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 16:17:15.711255 (+ 179us) client_negotiation.cc:240] Negotiation successful
0924 16:17:15.711700 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":332,"threads_started":1}
I20260924 16:17:15.714839 17509 heartbeater.cc:347] Connected to a master server at 127.29.234.62:43915
I20260924 16:17:15.715167 17509 heartbeater.cc:464] Registering TS with master...
I20260924 16:17:15.715929 17509 heartbeater.cc:511] Master 127.29.234.62:43915 requested a full tablet report, sending...
I20260924 16:17:15.718036 17320 ts_manager.cc:194] Registered new tserver with Master: 1d315e0deca04ed79705e9671e85f206 (127.29.234.1:40741)
I20260924 16:17:15.719501 17320 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.234.1:40271
W20260924 16:17:16.042326 17513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:17:16.042663 17513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:17:16.042820 17513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:17:16.050190 17513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:17:16.050311 17513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:17:16.050489 17513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.2
I20260924 16:17:16.059546 17513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:38859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266323169505-30632-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266323169505-30632-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.234.2
--webserver_port=0
--tserver_master_addrs=127.29.234.62:43915
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.2
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266323169505-30632-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:17:16.060936 17513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:17:16.063067 17513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:17:16.076205 17518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:16.076223 17519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:16.076227 17521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:17:16.077096 17513 server_base.cc:1034] running on GCE node
I20260924 16:17:16.077813 17513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:17:16.078775 17513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:17:16.080046 17513 hybrid_clock.cc:648] HybridClock initialized: now 1790266636080004 us; error 49 us; skew 500 ppm
I20260924 16:17:16.080407 17513 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 16:17:16.082686 17513 webserver.cc:460] Webserver started at http://127.29.234.2:34281/ using document root <none> and password file <none>
I20260924 16:17:16.083410 17513 fs_manager.cc:359] Metadata directory not provided
I20260924 16:17:16.083539 17513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:17:16.083854 17513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:17:16.086936 17513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266323169505-30632-0/minicluster-data/ts-1/data/instance:
uuid: "5ccc9cf5d94e46b9a21decd012bb405a"
format_stamp: "Formatted at 2026-09-24 16:17:16 on dist-test-slave-v5j9"
server_key: "581dd557a0f9ec917459e46d2a57f7dd"
server_key_iv: "8667479ffeff6aff257839d54a81039a"
server_key_version: "encryptionkey@0"
I20260924 16:17:16.087713 17513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266323169505-30632-0/minicluster-data/ts-1/wal/instance:
uuid: "5ccc9cf5d94e46b9a21decd012bb405a"
format_stamp: "Formatted at 2026-09-24 16:17:16 on dist-test-slave-v5j9"
server_key: "581dd557a0f9ec917459e46d2a57f7dd"
server_key_iv: "8667479ffeff6aff257839d54a81039a"
server_key_version: "encryptionkey@0"
I20260924 16:17:16.092723 17513 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 16:17:16.096100 17527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:17:16.097429 17513 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 16:17:16.097594 17513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266323169505-30632-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266323169505-30632-0/minicluster-data/ts-1/wal
uuid: "5ccc9cf5d94e46b9a21decd012bb405a"
format_stamp: "Formatted at 2026-09-24 16:17:16 on dist-test-slave-v5j9"
server_key: "581dd557a0f9ec917459e46d2a57f7dd"
server_key_iv: "8667479ffeff6aff257839d54a81039a"
server_key_version: "encryptionkey@0"
I20260924 16:17:16.097791 17513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266323169505-30632-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266323169505-30632-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266323169505-30632-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:17:16.116714 17513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:17:16.117743 17513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:17:16.118100 17513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:17:16.119244 17513 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:17:16.121055 17513 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:17:16.121160 17513 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:17:16.121246 17513 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:17:16.121325 17513 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:17:16.155318 17513 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.2:34977
I20260924 16:17:16.155391 17641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.2:34977 every 8 connection(s)
I20260924 16:17:16.156669 17513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
I20260924 16:17:16.159578 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 17513
I20260924 16:17:16.159761 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266323169505-30632-0/minicluster-data/ts-1/wal/instance
I20260924 16:17:16.160256 30632 external_mini_cluster.cc:1428] Setting key 7237ff7d8ad3c6bb5e73ce47007dddf7
I20260924 16:17:16.163592 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266323169505-30632-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266323169505-30632-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266323169505-30632-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.3:0
--local_ip_for_outbound_sockets=127.29.234.3
--webserver_interface=127.29.234.3
--webserver_port=0
--tserver_master_addrs=127.29.234.62:43915
--builtin_ntp_servers=127.29.234.20:38859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 16:17:16.171746 17511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:16.159383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.2:47033 (local address 127.29.234.62:43915)
0924 16:17:16.159607 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:16.159615 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:16.161402 (+ 1787us) server_negotiation.cc:407] Connection header received
0924 16:17:16.163841 (+ 2439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:16.163849 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:16.163935 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:16.164009 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:16.165780 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:16.166511 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:16.168941 (+ 2430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:16.169324 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:16.169469 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:16.171058 (+ 1589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:16.171063 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:16.171066 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:17:16.171084 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:16.171116 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:17:16.171128 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:16.171193 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:16.171443 (+ 250us) server_negotiation.cc:299] Negotiation successful
0924 16:17:16.171521 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 16:17:16.173302 17644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:16.160105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:43915 (local address 127.29.234.2:47033)
0924 16:17:16.161199 (+ 1094us) negotiation.cc:107] Waiting for socket to connect
0924 16:17:16.161248 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 16:17:16.163486 (+ 2238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:17:16.164553 (+ 1067us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:17:16.164567 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:17:16.164707 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:17:16.165577 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:16.165605 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:16.166683 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:16.166691 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:16.168795 (+ 2104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:16.168813 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:16.169593 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:16.169602 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:16.169950 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:16.170697 (+ 747us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:17:16.170703 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:17:16.170756 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:17:16.170868 (+ 112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:17:16.171223 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:17:16.171236 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:17:16.171247 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 16:17:16.171430 (+ 183us) client_negotiation.cc:240] Negotiation successful
0924 16:17:16.171871 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":331,"threads_started":1}
I20260924 16:17:16.174978 17642 heartbeater.cc:347] Connected to a master server at 127.29.234.62:43915
I20260924 16:17:16.175328 17642 heartbeater.cc:464] Registering TS with master...
I20260924 16:17:16.176260 17642 heartbeater.cc:511] Master 127.29.234.62:43915 requested a full tablet report, sending...
I20260924 16:17:16.177971 17320 ts_manager.cc:194] Registered new tserver with Master: 5ccc9cf5d94e46b9a21decd012bb405a (127.29.234.2:34977)
I20260924 16:17:16.178817 17320 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.234.2:47033
W20260924 16:17:16.495625 17645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:17:16.495947 17645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:17:16.496104 17645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:17:16.503566 17645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:17:16.503670 17645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:17:16.503846 17645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.3
I20260924 16:17:16.512964 17645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:38859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266323169505-30632-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266323169505-30632-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.234.3
--webserver_port=0
--tserver_master_addrs=127.29.234.62:43915
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.3
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266323169505-30632-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:17:16.514413 17645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:17:16.516213 17645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:17:16.529539 17650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:16.529593 17651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:16.529641 17653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:17:16.530886 17645 server_base.cc:1034] running on GCE node
I20260924 16:17:16.531512 17645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:17:16.532444 17645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:17:16.533644 17645 hybrid_clock.cc:648] HybridClock initialized: now 1790266636533596 us; error 41 us; skew 500 ppm
I20260924 16:17:16.533977 17645 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 16:17:16.536320 17645 webserver.cc:460] Webserver started at http://127.29.234.3:37027/ using document root <none> and password file <none>
I20260924 16:17:16.537045 17645 fs_manager.cc:359] Metadata directory not provided
I20260924 16:17:16.537159 17645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:17:16.537554 17645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:17:16.540525 17645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266323169505-30632-0/minicluster-data/ts-2/data/instance:
uuid: "0573dbb2dd284de19310fc6cc7c1f262"
format_stamp: "Formatted at 2026-09-24 16:17:16 on dist-test-slave-v5j9"
server_key: "a6fb2e79a1cfc64bed6766f00830e911"
server_key_iv: "bde89e31ed2bd4025cd759db173a0cae"
server_key_version: "encryptionkey@0"
I20260924 16:17:16.541460 17645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266323169505-30632-0/minicluster-data/ts-2/wal/instance:
uuid: "0573dbb2dd284de19310fc6cc7c1f262"
format_stamp: "Formatted at 2026-09-24 16:17:16 on dist-test-slave-v5j9"
server_key: "a6fb2e79a1cfc64bed6766f00830e911"
server_key_iv: "bde89e31ed2bd4025cd759db173a0cae"
server_key_version: "encryptionkey@0"
I20260924 16:17:16.546912 17645 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260924 16:17:16.550550 17659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:17:16.552006 17645 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:17:16.552182 17645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266323169505-30632-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266323169505-30632-0/minicluster-data/ts-2/wal
uuid: "0573dbb2dd284de19310fc6cc7c1f262"
format_stamp: "Formatted at 2026-09-24 16:17:16 on dist-test-slave-v5j9"
server_key: "a6fb2e79a1cfc64bed6766f00830e911"
server_key_iv: "bde89e31ed2bd4025cd759db173a0cae"
server_key_version: "encryptionkey@0"
I20260924 16:17:16.552412 17645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266323169505-30632-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266323169505-30632-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266323169505-30632-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:17:16.574411 17645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:17:16.575291 17645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:17:16.575555 17645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:17:16.576530 17645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:17:16.578003 17645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:17:16.578088 17645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:17:16.578171 17645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:17:16.578224 17645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:17:16.611596 17645 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.3:36003
I20260924 16:17:16.611688 17772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.3:36003 every 8 connection(s)
I20260924 16:17:16.612942 17645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
I20260924 16:17:16.613263 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 17645
I20260924 16:17:16.613497 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266323169505-30632-0/minicluster-data/ts-2/wal/instance
I20260924 16:17:16.614068 30632 external_mini_cluster.cc:1428] Setting key 8cd104538be5ec61c74d4cda221ac33b
I20260924 16:17:16.626394 17511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:16.615856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.3:35017 (local address 127.29.234.62:43915)
0924 16:17:16.616064 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:16.616070 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:16.617959 (+ 1889us) server_negotiation.cc:407] Connection header received
0924 16:17:16.619778 (+ 1819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:16.619784 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:16.619874 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:16.619998 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:16.621116 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:16.621942 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:16.623532 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:16.623950 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:16.624090 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:16.625533 (+ 1443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:16.625538 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:16.625542 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:17:16.625560 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:16.625599 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:17:16.625611 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:16.625711 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:16.625933 (+ 222us) server_negotiation.cc:299] Negotiation successful
0924 16:17:16.626057 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 16:17:16.627372 17775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:16.616619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:43915 (local address 127.29.234.3:35017)
0924 16:17:16.617809 (+ 1190us) negotiation.cc:107] Waiting for socket to connect
0924 16:17:16.617851 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 16:17:16.619507 (+ 1656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:17:16.620246 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:17:16.620255 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:17:16.620344 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:17:16.620948 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:16.620966 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:16.622116 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:16.622120 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:16.623376 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:16.623388 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:16.624188 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:16.624195 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:16.624429 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:16.625059 (+ 630us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:17:16.625061 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:17:16.625150 (+ 89us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:17:16.625275 (+ 125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:17:16.625711 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:17:16.625724 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:17:16.625735 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 16:17:16.625917 (+ 182us) client_negotiation.cc:240] Negotiation successful
0924 16:17:16.626345 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":357,"threads_started":1}
I20260924 16:17:16.628815 17773 heartbeater.cc:347] Connected to a master server at 127.29.234.62:43915
I20260924 16:17:16.629153 17773 heartbeater.cc:464] Registering TS with master...
I20260924 16:17:16.630070 17773 heartbeater.cc:511] Master 127.29.234.62:43915 requested a full tablet report, sending...
I20260924 16:17:16.631314 17320 ts_manager.cc:194] Registered new tserver with Master: 0573dbb2dd284de19310fc6cc7c1f262 (127.29.234.3:36003)
I20260924 16:17:16.632045 17320 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.234.3:35017
I20260924 16:17:16.642406 30632 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:17:16.652335 17511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:16.646535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56048 (local address 127.29.234.62:43915)
0924 16:17:16.646761 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:16.646766 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:16.647053 (+ 287us) server_negotiation.cc:407] Connection header received
0924 16:17:16.647208 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:16.647213 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:16.647306 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:16.647392 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:16.647871 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:16.648577 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:16.649984 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:16.650405 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:16.650499 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:16.651556 (+ 1057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:16.651561 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:16.651564 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:17:16.651580 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:16.651610 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:17:16.651622 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:16.651838 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:16.651991 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 16:17:16.652035 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 16:17:16.656356 17320 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56048:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:17:16.664268 17320 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:17:16.688221 17787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:16.675967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.2:34977 (local address 127.0.0.1:60252)
0924 16:17:16.678635 (+ 2668us) negotiation.cc:107] Waiting for socket to connect
0924 16:17:16.678692 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 16:17:16.678906 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:17:16.680778 (+ 1872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:17:16.680786 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:17:16.680815 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:17:16.681388 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:16.681407 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:16.683306 (+ 1899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:16.683338 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:16.685620 (+ 2282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:16.685634 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:16.687283 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:16.687290 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:16.687515 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:16.687569 (+ 54us) client_negotiation.cc:769] Sending connection context
0924 16:17:16.687691 (+ 122us) client_negotiation.cc:240] Negotiation successful
0924 16:17:16.687865 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2356,"thread_start_us":7074,"threads_started":1}
I20260924 16:17:16.688570 17786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:16.676112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60252 (local address 127.29.234.2:34977)
0924 16:17:16.679876 (+ 3764us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:16.679884 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:16.679915 (+ 31us) server_negotiation.cc:407] Connection header received
0924 16:17:16.680020 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:16.680027 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:16.680289 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:16.680484 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:17:16.681608 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:16.683134 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:16.685862 (+ 2728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:16.687083 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:16.687232 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:16.687479 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:16.687784 (+ 305us) server_negotiation.cc:299] Negotiation successful
0924 16:17:16.687957 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3632,"thread_start_us":3541,"threads_started":1}
I20260924 16:17:16.689388 17789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:16.676135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.3:36003 (local address 127.0.0.1:44266)
0924 16:17:16.681328 (+ 5193us) negotiation.cc:107] Waiting for socket to connect
0924 16:17:16.681371 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 16:17:16.681495 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:17:16.682409 (+ 914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:17:16.682414 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:17:16.682438 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:17:16.682813 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:16.682824 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:16.684654 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:16.684660 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:16.686815 (+ 2155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:16.686827 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:16.688628 (+ 1801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:16.688635 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:16.688850 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:16.688895 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 16:17:16.688958 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 16:17:16.689085 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5029,"thread_start_us":3721,"threads_started":1}
I20260924 16:17:16.690608 17788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:16.676603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44266 (local address 127.29.234.3:36003)
0924 16:17:16.677899 (+ 1296us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:16.677912 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:16.681617 (+ 3705us) server_negotiation.cc:407] Connection header received
0924 16:17:16.681740 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:16.681748 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:16.681981 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:16.682162 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:17:16.683035 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:16.684514 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:16.687004 (+ 2490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:16.688383 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:16.689719 (+ 1336us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:16.690004 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:16.690080 (+ 76us) server_negotiation.cc:299] Negotiation successful
0924 16:17:16.690240 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1091,"thread_start_us":930,"threads_started":1}
I20260924 16:17:16.691419 17785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:16.678095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:40741 (local address 127.0.0.1:48580)
0924 16:17:16.683298 (+ 5203us) negotiation.cc:107] Waiting for socket to connect
0924 16:17:16.683315 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:17:16.683512 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:17:16.684122 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:17:16.684127 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:17:16.684152 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:17:16.684549 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:16.684561 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:16.686551 (+ 1990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:16.686558 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:16.688589 (+ 2031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:16.688611 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:16.690502 (+ 1891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:16.690509 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:16.690831 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:16.690870 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 16:17:16.690974 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 16:17:16.691090 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5074,"mutex_wait_us":1,"thread_start_us":2918,"threads_started":1}
I20260924 16:17:16.691578 17790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:16.678879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48580 (local address 127.29.234.1:40741)
0924 16:17:16.680864 (+ 1985us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:16.680874 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:16.683402 (+ 2528us) server_negotiation.cc:407] Connection header received
0924 16:17:16.683707 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:16.683715 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:16.683922 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:16.684111 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:17:16.684739 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:16.686398 (+ 1659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:16.688821 (+ 2423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:16.690304 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:16.690444 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:16.690680 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:16.691035 (+ 355us) server_negotiation.cc:299] Negotiation successful
0924 16:17:16.691194 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1817,"thread_start_us":1690,"threads_started":1}
I20260924 16:17:16.692906 17576 tablet_service.cc:1467] Processing CreateTablet for tablet 13b6edf4f5314f0289dab5ab024ad243 (DEFAULT_TABLE table=test-table [id=e5b4af821fd4474485ba2ffc3d541bb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:17:16.693948 17707 tablet_service.cc:1467] Processing CreateTablet for tablet 13b6edf4f5314f0289dab5ab024ad243 (DEFAULT_TABLE table=test-table [id=e5b4af821fd4474485ba2ffc3d541bb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:17:16.694486 17576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13b6edf4f5314f0289dab5ab024ad243. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:17:16.695381 17707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13b6edf4f5314f0289dab5ab024ad243. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:17:16.695183 17443 tablet_service.cc:1467] Processing CreateTablet for tablet 13b6edf4f5314f0289dab5ab024ad243 (DEFAULT_TABLE table=test-table [id=e5b4af821fd4474485ba2ffc3d541bb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:17:16.696497 17443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13b6edf4f5314f0289dab5ab024ad243. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:17:16.712445 17793 tablet_bootstrap.cc:492] T 13b6edf4f5314f0289dab5ab024ad243 P 1d315e0deca04ed79705e9671e85f206: Bootstrap starting.
I20260924 16:17:16.715716 17792 tablet_bootstrap.cc:492] T 13b6edf4f5314f0289dab5ab024ad243 P 0573dbb2dd284de19310fc6cc7c1f262: Bootstrap starting.
I20260924 16:17:16.717623 17791 tablet_bootstrap.cc:492] T 13b6edf4f5314f0289dab5ab024ad243 P 5ccc9cf5d94e46b9a21decd012bb405a: Bootstrap starting.
I20260924 16:17:16.718384 17793 tablet_bootstrap.cc:654] T 13b6edf4f5314f0289dab5ab024ad243 P 1d315e0deca04ed79705e9671e85f206: Neither blocks nor log segments found. Creating new log.
I20260924 16:17:16.719686 17793 log.cc:824] T 13b6edf4f5314f0289dab5ab024ad243 P 1d315e0deca04ed79705e9671e85f206: Log is configured to *not* fsync() on all Append() calls
I20260924 16:17:16.720081 17792 tablet_bootstrap.cc:654] T 13b6edf4f5314f0289dab5ab024ad243 P 0573dbb2dd284de19310fc6cc7c1f262: Neither blocks nor log segments found. Creating new log.
I20260924 16:17:16.720587 17791 tablet_bootstrap.cc:654] T 13b6edf4f5314f0289dab5ab024ad243 P 5ccc9cf5d94e46b9a21decd012bb405a: Neither blocks nor log segments found. Creating new log.
I20260924 16:17:16.721279 17792 log.cc:824] T 13b6edf4f5314f0289dab5ab024ad243 P 0573dbb2dd284de19310fc6cc7c1f262: Log is configured to *not* fsync() on all Append() calls
I20260924 16:17:16.721742 17791 log.cc:824] T 13b6edf4f5314f0289dab5ab024ad243 P 5ccc9cf5d94e46b9a21decd012bb405a: Log is configured to *not* fsync() on all Append() calls
I20260924 16:17:16.722584 17509 heartbeater.cc:503] Master 127.29.234.62:43915 was elected leader, sending a full tablet report...
I20260924 16:17:16.723232 17793 tablet_bootstrap.cc:492] T 13b6edf4f5314f0289dab5ab024ad243 P 1d315e0deca04ed79705e9671e85f206: No bootstrap required, opened a new log
I20260924 16:17:16.723598 17793 ts_tablet_manager.cc:1397] T 13b6edf4f5314f0289dab5ab024ad243 P 1d315e0deca04ed79705e9671e85f206: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260924 16:17:16.725929 17792 tablet_bootstrap.cc:492] T 13b6edf4f5314f0289dab5ab024ad243 P 0573dbb2dd284de19310fc6cc7c1f262: No bootstrap required, opened a new log
I20260924 16:17:16.726158 17791 tablet_bootstrap.cc:492] T 13b6edf4f5314f0289dab5ab024ad243 P 5ccc9cf5d94e46b9a21decd012bb405a: No bootstrap required, opened a new log
I20260924 16:17:16.726408 17791 ts_tablet_manager.cc:1397] T 13b6edf4f5314f0289dab5ab024ad243 P 5ccc9cf5d94e46b9a21decd012bb405a: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260924 16:17:16.726446 17792 ts_tablet_manager.cc:1397] T 13b6edf4f5314f0289dab5ab024ad243 P 0573dbb2dd284de19310fc6cc7c1f262: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260924 16:17:16.733741 17791 raft_consensus.cc:348] T 13b6edf4f5314f0289dab5ab024ad243 P 5ccc9cf5d94e46b9a21decd012bb405a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ccc9cf5d94e46b9a21decd012bb405a" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 34977 } } peers { permanent_uuid: "0573dbb2dd284de19310fc6cc7c1f262" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 36003 } } peers { permanent_uuid: "1d315e0deca04ed79705e9671e85f206" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 40741 } }
I20260924 16:17:16.734123 17791 raft_consensus.cc:374] T 13b6edf4f5314f0289dab5ab024ad243 P 5ccc9cf5d94e46b9a21decd012bb405a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:17:16.734259 17791 raft_consensus.cc:729] T 13b6edf4f5314f0289dab5ab024ad243 P 5ccc9cf5d94e46b9a21decd012bb405a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ccc9cf5d94e46b9a21decd012bb405a, State: Initialized, Role: FOLLOWER
I20260924 16:17:16.734514 17793 raft_consensus.cc:348] T 13b6edf4f5314f0289dab5ab024ad243 P 1d315e0deca04ed79705e9671e85f206 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ccc9cf5d94e46b9a21decd012bb405a" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 34977 } } peers { permanent_uuid: "0573dbb2dd284de19310fc6cc7c1f262" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 36003 } } peers { permanent_uuid: "1d315e0deca04ed79705e9671e85f206" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 40741 } }
I20260924 16:17:16.734951 17793 raft_consensus.cc:374] T 13b6edf4f5314f0289dab5ab024ad243 P 1d315e0deca04ed79705e9671e85f206 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:17:16.735054 17793 raft_consensus.cc:729] T 13b6edf4f5314f0289dab5ab024ad243 P 1d315e0deca04ed79705e9671e85f206 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d315e0deca04ed79705e9671e85f206, State: Initialized, Role: FOLLOWER
I20260924 16:17:16.734910 17791 consensus_queue.cc:260] T 13b6edf4f5314f0289dab5ab024ad243 P 5ccc9cf5d94e46b9a21decd012bb405a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ccc9cf5d94e46b9a21decd012bb405a" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 34977 } } peers { permanent_uuid: "0573dbb2dd284de19310fc6cc7c1f262" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 36003 } } peers { permanent_uuid: "1d315e0deca04ed79705e9671e85f206" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 40741 } }
I20260924 16:17:16.735638 17793 consensus_queue.cc:260] T 13b6edf4f5314f0289dab5ab024ad243 P 1d315e0deca04ed79705e9671e85f206 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ccc9cf5d94e46b9a21decd012bb405a" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 34977 } } peers { permanent_uuid: "0573dbb2dd284de19310fc6cc7c1f262" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 36003 } } peers { permanent_uuid: "1d315e0deca04ed79705e9671e85f206" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 40741 } }
I20260924 16:17:16.736913 17792 raft_consensus.cc:348] T 13b6edf4f5314f0289dab5ab024ad243 P 0573dbb2dd284de19310fc6cc7c1f262 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ccc9cf5d94e46b9a21decd012bb405a" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 34977 } } peers { permanent_uuid: "0573dbb2dd284de19310fc6cc7c1f262" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 36003 } } peers { permanent_uuid: "1d315e0deca04ed79705e9671e85f206" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 40741 } }
I20260924 16:17:16.737421 17792 raft_consensus.cc:374] T 13b6edf4f5314f0289dab5ab024ad243 P 0573dbb2dd284de19310fc6cc7c1f262 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:17:16.737577 17792 raft_consensus.cc:729] T 13b6edf4f5314f0289dab5ab024ad243 P 0573dbb2dd284de19310fc6cc7c1f262 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0573dbb2dd284de19310fc6cc7c1f262, State: Initialized, Role: FOLLOWER
I20260924 16:17:16.737608 17793 ts_tablet_manager.cc:1428] T 13b6edf4f5314f0289dab5ab024ad243 P 1d315e0deca04ed79705e9671e85f206: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260924 16:17:16.738361 17792 consensus_queue.cc:260] T 13b6edf4f5314f0289dab5ab024ad243 P 0573dbb2dd284de19310fc6cc7c1f262 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ccc9cf5d94e46b9a21decd012bb405a" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 34977 } } peers { permanent_uuid: "0573dbb2dd284de19310fc6cc7c1f262" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 36003 } } peers { permanent_uuid: "1d315e0deca04ed79705e9671e85f206" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 40741 } }
I20260924 16:17:16.739693 17773 heartbeater.cc:503] Master 127.29.234.62:43915 was elected leader, sending a full tablet report...
I20260924 16:17:16.740329 17792 ts_tablet_manager.cc:1428] T 13b6edf4f5314f0289dab5ab024ad243 P 0573dbb2dd284de19310fc6cc7c1f262: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260924 16:17:16.741245 17642 heartbeater.cc:503] Master 127.29.234.62:43915 was elected leader, sending a full tablet report...
I20260924 16:17:16.741935 17791 ts_tablet_manager.cc:1428] T 13b6edf4f5314f0289dab5ab024ad243 P 5ccc9cf5d94e46b9a21decd012bb405a: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
I20260924 16:17:16.792265 17797 raft_consensus.cc:482] T 13b6edf4f5314f0289dab5ab024ad243 P 5ccc9cf5d94e46b9a21decd012bb405a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:17:16.792584 17797 raft_consensus.cc:504] T 13b6edf4f5314f0289dab5ab024ad243 P 5ccc9cf5d94e46b9a21decd012bb405a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ccc9cf5d94e46b9a21decd012bb405a" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 34977 } } peers { permanent_uuid: "0573dbb2dd284de19310fc6cc7c1f262" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 36003 } } peers { permanent_uuid: "1d315e0deca04ed79705e9671e85f206" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 40741 } }
I20260924 16:17:16.794458 17797 leader_election.cc:288] T 13b6edf4f5314f0289dab5ab024ad243 P 5ccc9cf5d94e46b9a21decd012bb405a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0573dbb2dd284de19310fc6cc7c1f262 (127.29.234.3:36003), 1d315e0deca04ed79705e9671e85f206 (127.29.234.1:40741)
I20260924 16:17:16.801452 17788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:16.794918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.2:55077 (local address 127.29.234.3:36003)
0924 16:17:16.795126 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:16.795134 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:16.795869 (+ 735us) server_negotiation.cc:407] Connection header received
0924 16:17:16.796098 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:16.796102 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:16.796169 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:16.796269 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:17:16.796839 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:16.797629 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:16.799599 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:16.800378 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:16.800573 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:16.800652 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:16.801084 (+ 432us) server_negotiation.cc:299] Negotiation successful
0924 16:17:16.801214 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 16:17:16.801460 17790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:16.795122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.2:48513 (local address 127.29.234.1:40741)
0924 16:17:16.795383 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:16.795392 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:16.796021 (+ 629us) server_negotiation.cc:407] Connection header received
0924 16:17:16.796229 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:16.796233 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:16.796300 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:16.796399 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:17:16.797011 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:16.797735 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:16.799616 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:16.800375 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:16.800548 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:16.800625 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:16.801010 (+ 385us) server_negotiation.cc:299] Negotiation successful
0924 16:17:16.801158 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 16:17:16.801512 17800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:16.794901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.3:36003 (local address 127.29.234.2:55077)
0924 16:17:16.795743 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0924 16:17:16.795762 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 16:17:16.795944 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:17:16.796285 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:17:16.796289 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:17:16.796312 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:17:16.796690 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:16.796699 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:16.797754 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:16.797763 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:16.799404 (+ 1641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:16.799416 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:16.800629 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:16.800637 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:16.800857 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:16.800880 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 16:17:16.801031 (+ 151us) client_negotiation.cc:240] Negotiation successful
0924 16:17:16.801152 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"mutex_wait_us":98,"thread_start_us":474,"threads_started":1}
I20260924 16:17:16.802060 17801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:16.794945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:40741 (local address 127.29.234.2:48513)
0924 16:17:16.795912 (+ 967us) negotiation.cc:107] Waiting for socket to connect
0924 16:17:16.795925 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 16:17:16.796072 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:17:16.796418 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:17:16.796422 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:17:16.796460 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:17:16.796858 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:16.796866 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:16.797879 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:16.797884 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:16.799409 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:16.799420 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:16.800585 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:16.800591 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:16.800819 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:16.800847 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 16:17:16.800958 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 16:17:16.801099 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":856,"mutex_wait_us":42,"thread_start_us":635,"threads_started":1}
I20260924 16:17:16.802955 17727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13b6edf4f5314f0289dab5ab024ad243" candidate_uuid: "5ccc9cf5d94e46b9a21decd012bb405a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0573dbb2dd284de19310fc6cc7c1f262" is_pre_election: true
I20260924 16:17:16.803103 17463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13b6edf4f5314f0289dab5ab024ad243" candidate_uuid: "5ccc9cf5d94e46b9a21decd012bb405a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d315e0deca04ed79705e9671e85f206" is_pre_election: true
I20260924 16:17:16.803598 17727 raft_consensus.cc:2454] T 13b6edf4f5314f0289dab5ab024ad243 P 0573dbb2dd284de19310fc6cc7c1f262 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ccc9cf5d94e46b9a21decd012bb405a in term 0.
I20260924 16:17:16.803622 17463 raft_consensus.cc:2454] T 13b6edf4f5314f0289dab5ab024ad243 P 1d315e0deca04ed79705e9671e85f206 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ccc9cf5d94e46b9a21decd012bb405a in term 0.
I20260924 16:17:16.804356 17530 leader_election.cc:302] T 13b6edf4f5314f0289dab5ab024ad243 P 5ccc9cf5d94e46b9a21decd012bb405a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0573dbb2dd284de19310fc6cc7c1f262, 1d315e0deca04ed79705e9671e85f206, 5ccc9cf5d94e46b9a21decd012bb405a; no voters:
I20260924 16:17:16.804814 17797 raft_consensus.cc:2781] T 13b6edf4f5314f0289dab5ab024ad243 P 5ccc9cf5d94e46b9a21decd012bb405a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:17:16.804958 17797 raft_consensus.cc:482] T 13b6edf4f5314f0289dab5ab024ad243 P 5ccc9cf5d94e46b9a21decd012bb405a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:17:16.805068 17797 raft_consensus.cc:3037] T 13b6edf4f5314f0289dab5ab024ad243 P 5ccc9cf5d94e46b9a21decd012bb405a [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:17:16.807334 17797 raft_consensus.cc:504] T 13b6edf4f5314f0289dab5ab024ad243 P 5ccc9cf5d94e46b9a21decd012bb405a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ccc9cf5d94e46b9a21decd012bb405a" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 34977 } } peers { permanent_uuid: "0573dbb2dd284de19310fc6cc7c1f262" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 36003 } } peers { permanent_uuid: "1d315e0deca04ed79705e9671e85f206" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 40741 } }
I20260924 16:17:16.807945 17797 leader_election.cc:288] T 13b6edf4f5314f0289dab5ab024ad243 P 5ccc9cf5d94e46b9a21decd012bb405a [CANDIDATE]: Term 1 election: Requested vote from peers 0573dbb2dd284de19310fc6cc7c1f262 (127.29.234.3:36003), 1d315e0deca04ed79705e9671e85f206 (127.29.234.1:40741)
I20260924 16:17:16.808468 17727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13b6edf4f5314f0289dab5ab024ad243" candidate_uuid: "5ccc9cf5d94e46b9a21decd012bb405a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0573dbb2dd284de19310fc6cc7c1f262"
I20260924 16:17:16.808470 17463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13b6edf4f5314f0289dab5ab024ad243" candidate_uuid: "5ccc9cf5d94e46b9a21decd012bb405a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d315e0deca04ed79705e9671e85f206"
I20260924 16:17:16.808687 17727 raft_consensus.cc:3037] T 13b6edf4f5314f0289dab5ab024ad243 P 0573dbb2dd284de19310fc6cc7c1f262 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:17:16.808687 17463 raft_consensus.cc:3037] T 13b6edf4f5314f0289dab5ab024ad243 P 1d315e0deca04ed79705e9671e85f206 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:17:16.811944 17463 raft_consensus.cc:2454] T 13b6edf4f5314f0289dab5ab024ad243 P 1d315e0deca04ed79705e9671e85f206 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ccc9cf5d94e46b9a21decd012bb405a in term 1.
I20260924 16:17:16.811942 17727 raft_consensus.cc:2454] T 13b6edf4f5314f0289dab5ab024ad243 P 0573dbb2dd284de19310fc6cc7c1f262 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ccc9cf5d94e46b9a21decd012bb405a in term 1.
I20260924 16:17:16.812402 17528 leader_election.cc:302] T 13b6edf4f5314f0289dab5ab024ad243 P 5ccc9cf5d94e46b9a21decd012bb405a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0573dbb2dd284de19310fc6cc7c1f262, 1d315e0deca04ed79705e9671e85f206, 5ccc9cf5d94e46b9a21decd012bb405a; no voters:
I20260924 16:17:16.812769 17797 raft_consensus.cc:2781] T 13b6edf4f5314f0289dab5ab024ad243 P 5ccc9cf5d94e46b9a21decd012bb405a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:17:16.813220 17797 raft_consensus.cc:686] T 13b6edf4f5314f0289dab5ab024ad243 P 5ccc9cf5d94e46b9a21decd012bb405a [term 1 LEADER]: Becoming Leader. State: Replica: 5ccc9cf5d94e46b9a21decd012bb405a, State: Running, Role: LEADER
I20260924 16:17:16.813712 17797 consensus_queue.cc:237] T 13b6edf4f5314f0289dab5ab024ad243 P 5ccc9cf5d94e46b9a21decd012bb405a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ccc9cf5d94e46b9a21decd012bb405a" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 34977 } } peers { permanent_uuid: "0573dbb2dd284de19310fc6cc7c1f262" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 36003 } } peers { permanent_uuid: "1d315e0deca04ed79705e9671e85f206" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 40741 } }
I20260924 16:17:16.818572 17320 catalog_manager.cc:5495] T 13b6edf4f5314f0289dab5ab024ad243 P 5ccc9cf5d94e46b9a21decd012bb405a reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ccc9cf5d94e46b9a21decd012bb405a (127.29.234.2). New cstate: current_term: 1 leader_uuid: "5ccc9cf5d94e46b9a21decd012bb405a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ccc9cf5d94e46b9a21decd012bb405a" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 34977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0573dbb2dd284de19310fc6cc7c1f262" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 36003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d315e0deca04ed79705e9671e85f206" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 40741 } health_report { overall_health: UNKNOWN } } }
I20260924 16:17:16.844928 17786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:16.839270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60262 (local address 127.29.234.2:34977)
0924 16:17:16.839479 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:16.839485 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:16.839506 (+ 21us) server_negotiation.cc:407] Connection header received
0924 16:17:16.839698 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:16.839702 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:16.839793 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:16.839946 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:17:16.840347 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:16.841096 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:16.842569 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:16.842989 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:16.843131 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:16.843749 (+ 618us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:17:16.844332 (+ 583us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:17:16.844522 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:16.844642 (+ 120us) server_negotiation.cc:299] Negotiation successful
0924 16:17:16.844686 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 16:17:16.852999 17463 raft_consensus.cc:1261] T 13b6edf4f5314f0289dab5ab024ad243 P 1d315e0deca04ed79705e9671e85f206 [term 1 FOLLOWER]: Refusing update from remote peer 5ccc9cf5d94e46b9a21decd012bb405a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:17:16.853219 17727 raft_consensus.cc:1261] T 13b6edf4f5314f0289dab5ab024ad243 P 0573dbb2dd284de19310fc6cc7c1f262 [term 1 FOLLOWER]: Refusing update from remote peer 5ccc9cf5d94e46b9a21decd012bb405a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:17:16.854038 17802 consensus_queue.cc:1035] T 13b6edf4f5314f0289dab5ab024ad243 P 5ccc9cf5d94e46b9a21decd012bb405a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d315e0deca04ed79705e9671e85f206" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 40741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:17:16.854398 17797 consensus_queue.cc:1035] T 13b6edf4f5314f0289dab5ab024ad243 P 5ccc9cf5d94e46b9a21decd012bb405a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0573dbb2dd284de19310fc6cc7c1f262" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 36003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 16:17:16.864830 17774 tablet.cc:2369] T 13b6edf4f5314f0289dab5ab024ad243 P 0573dbb2dd284de19310fc6cc7c1f262: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:17:16.866765 17805 mvcc.cc:205] Tried to move back new op lower bound from 7332932144540475392 to 7332932144392921088. Current Snapshot: MvccSnapshot[applied={T|T < 7332932144540475392}]
I20260924 16:17:16.872272 17806 mvcc.cc:205] Tried to move back new op lower bound from 7332932144540475392 to 7332932144392921088. Current Snapshot: MvccSnapshot[applied={T|T < 7332932144540475392}]
I20260924 16:17:16.883929 17319 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56048:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:17:16.884205 17319 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56048:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:17:16.888607 17319 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 204b69aeb9eb425e93eb446e45ef6342: Sending DeleteTablet for 3 replicas of tablet 13b6edf4f5314f0289dab5ab024ad243
I20260924 16:17:16.889957 17707 tablet_service.cc:1514] Processing DeleteTablet for tablet 13b6edf4f5314f0289dab5ab024ad243 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:17:16 UTC) from {username='slave'} at 127.0.0.1:44266
I20260924 16:17:16.890144 17443 tablet_service.cc:1514] Processing DeleteTablet for tablet 13b6edf4f5314f0289dab5ab024ad243 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:17:16 UTC) from {username='slave'} at 127.0.0.1:48580
I20260924 16:17:16.890480 17576 tablet_service.cc:1514] Processing DeleteTablet for tablet 13b6edf4f5314f0289dab5ab024ad243 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:17:16 UTC) from {username='slave'} at 127.0.0.1:60252
I20260924 16:17:16.891247 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 17380
I20260924 16:17:16.902697 17814 tablet_replica.cc:330] T 13b6edf4f5314f0289dab5ab024ad243 P 0573dbb2dd284de19310fc6cc7c1f262: stopping tablet replica
I20260924 16:17:16.903427 17814 raft_consensus.cc:2229] T 13b6edf4f5314f0289dab5ab024ad243 P 0573dbb2dd284de19310fc6cc7c1f262 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:17:16.904098 17814 raft_consensus.cc:2258] T 13b6edf4f5314f0289dab5ab024ad243 P 0573dbb2dd284de19310fc6cc7c1f262 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:17:16.905071 17815 tablet_replica.cc:330] T 13b6edf4f5314f0289dab5ab024ad243 P 1d315e0deca04ed79705e9671e85f206: stopping tablet replica
I20260924 16:17:16.905613 17815 raft_consensus.cc:2229] T 13b6edf4f5314f0289dab5ab024ad243 P 1d315e0deca04ed79705e9671e85f206 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:17:16.906140 17815 raft_consensus.cc:2258] T 13b6edf4f5314f0289dab5ab024ad243 P 1d315e0deca04ed79705e9671e85f206 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:17:16.907414 17814 ts_tablet_manager.cc:1905] T 13b6edf4f5314f0289dab5ab024ad243 P 0573dbb2dd284de19310fc6cc7c1f262: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:17:16.908298 17815 ts_tablet_manager.cc:1905] T 13b6edf4f5314f0289dab5ab024ad243 P 1d315e0deca04ed79705e9671e85f206: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:17:16.914433 17814 ts_tablet_manager.cc:1918] T 13b6edf4f5314f0289dab5ab024ad243 P 0573dbb2dd284de19310fc6cc7c1f262: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:17:16.914610 17814 log.cc:1196] T 13b6edf4f5314f0289dab5ab024ad243 P 0573dbb2dd284de19310fc6cc7c1f262: Deleting WAL directory at /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266323169505-30632-0/minicluster-data/ts-2/wal/wals/13b6edf4f5314f0289dab5ab024ad243
I20260924 16:17:16.915153 17814 ts_tablet_manager.cc:1939] T 13b6edf4f5314f0289dab5ab024ad243 P 0573dbb2dd284de19310fc6cc7c1f262: Deleting consensus metadata
I20260924 16:17:16.915357 17815 ts_tablet_manager.cc:1918] T 13b6edf4f5314f0289dab5ab024ad243 P 1d315e0deca04ed79705e9671e85f206: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:17:16.915509 17815 log.cc:1196] T 13b6edf4f5314f0289dab5ab024ad243 P 1d315e0deca04ed79705e9671e85f206: Deleting WAL directory at /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266323169505-30632-0/minicluster-data/ts-0/wal/wals/13b6edf4f5314f0289dab5ab024ad243
I20260924 16:17:16.916019 17815 ts_tablet_manager.cc:1939] T 13b6edf4f5314f0289dab5ab024ad243 P 1d315e0deca04ed79705e9671e85f206: Deleting consensus metadata
I20260924 16:17:16.916728 17790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:16.893078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48582 (local address 127.29.234.1:40741)
0924 16:17:16.893311 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:16.893321 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:16.906104 (+ 12783us) server_negotiation.cc:407] Connection header received
0924 16:17:16.906501 (+ 397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:16.906507 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:16.906633 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:16.906714 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:16.907892 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:16.909062 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:16.912257 (+ 3195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:16.912966 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:16.913080 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:16.914720 (+ 1640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:16.914733 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:16.914754 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:17:16.915600 (+ 846us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:16.915667 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:17:16.915695 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:16.915785 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:16.916230 (+ 445us) server_negotiation.cc:299] Negotiation successful
0924 16:17:16.916356 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 16:17:16.917122 17304 catalog_manager.cc:4841] TS 0573dbb2dd284de19310fc6cc7c1f262 (127.29.234.3:36003): tablet 13b6edf4f5314f0289dab5ab024ad243 (table test-table [id=e5b4af821fd4474485ba2ffc3d541bb2]) successfully deleted
I20260924 16:17:16.918030 17306 catalog_manager.cc:4841] TS 1d315e0deca04ed79705e9671e85f206 (127.29.234.1:40741): tablet 13b6edf4f5314f0289dab5ab024ad243 (table test-table [id=e5b4af821fd4474485ba2ffc3d541bb2]) successfully deleted
I20260924 16:17:16.918133 17504 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:17:16.917331 17816 tablet_replica.cc:330] T 13b6edf4f5314f0289dab5ab024ad243 P 5ccc9cf5d94e46b9a21decd012bb405a: stopping tablet replica
I20260924 16:17:16.918779 17816 raft_consensus.cc:2229] T 13b6edf4f5314f0289dab5ab024ad243 P 5ccc9cf5d94e46b9a21decd012bb405a [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:17:16.919970 17816 raft_consensus.cc:2258] T 13b6edf4f5314f0289dab5ab024ad243 P 5ccc9cf5d94e46b9a21decd012bb405a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:17:16.923118 17816 ts_tablet_manager.cc:1905] T 13b6edf4f5314f0289dab5ab024ad243 P 5ccc9cf5d94e46b9a21decd012bb405a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:17:16.928705 17816 ts_tablet_manager.cc:1918] T 13b6edf4f5314f0289dab5ab024ad243 P 5ccc9cf5d94e46b9a21decd012bb405a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:17:16.928902 17816 log.cc:1196] T 13b6edf4f5314f0289dab5ab024ad243 P 5ccc9cf5d94e46b9a21decd012bb405a: Deleting WAL directory at /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790266323169505-30632-0/minicluster-data/ts-1/wal/wals/13b6edf4f5314f0289dab5ab024ad243
I20260924 16:17:16.929504 17816 ts_tablet_manager.cc:1939] T 13b6edf4f5314f0289dab5ab024ad243 P 5ccc9cf5d94e46b9a21decd012bb405a: Deleting consensus metadata
I20260924 16:17:16.931109 17305 catalog_manager.cc:4841] TS 5ccc9cf5d94e46b9a21decd012bb405a (127.29.234.2:34977): tablet 13b6edf4f5314f0289dab5ab024ad243 (table test-table [id=e5b4af821fd4474485ba2ffc3d541bb2]) successfully deleted
I20260924 16:17:17.023924 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 17380
I20260924 16:17:17.050760 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 17513
I20260924 16:17:17.058328 17786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:17.051641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60266 (local address 127.29.234.2:34977)
0924 16:17:17.051901 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:17.051908 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:17.051927 (+ 19us) server_negotiation.cc:407] Connection header received
0924 16:17:17.052029 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:17.052037 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:17.052121 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:17.052252 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:17.052686 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:17.053551 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:17.054818 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:17.055253 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:17.055387 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:17.056593 (+ 1206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:17.056605 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:17.056623 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:17:17.057518 (+ 895us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:17.057568 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:17:17.057588 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:17.057739 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:17.057871 (+ 132us) server_negotiation.cc:299] Negotiation successful
0924 16:17:17.057971 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
I20260924 16:17:17.059293 17637 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:17:17.171075 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 17513
I20260924 16:17:17.190577 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 17645
I20260924 16:17:17.205219 17788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:17.191330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44280 (local address 127.29.234.3:36003)
0924 16:17:17.191520 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:17.191526 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:17.191624 (+ 98us) server_negotiation.cc:407] Connection header received
0924 16:17:17.191834 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:17.191846 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:17.191927 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:17.192045 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:17.192442 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:17.193179 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:17.194432 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:17.194814 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:17.194949 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:17.203780 (+ 8831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:17.203793 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:17.203814 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:17:17.204418 (+ 604us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:17.204471 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:17:17.204493 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:17.204638 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:17.204776 (+ 138us) server_negotiation.cc:299] Negotiation successful
0924 16:17:17.204902 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 16:17:17.205909 17768 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:17:17.293748 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 17645
I20260924 16:17:17.310962 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 17289
I20260924 16:17:17.312052 17351 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:17:17.420590 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 17289
2026-09-24T16:17:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2821 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:17:17 dist-test-slave-v5j9 krb5kdc[17824](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:17:17 dist-test-slave-v5j9 krb5kdc[17824](info): set up 2 sockets
Sep 24 16:17:17 dist-test-slave-v5j9 krb5kdc[17824](info): commencing operation
krb5kdc: starting...
W20260924 16:17:18.528975 30632 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Sep 24 16:17:18 dist-test-slave-v5j9 krb5kdc[17824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:17:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:17:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.234.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab.
Entry for principal kudu/127.29.234.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab.
I20260924 16:17:18.790933 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:34075
--webserver_interface=127.29.234.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab
--principal=kudu/127.29.234.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:42195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.234.62:34075
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:17:19.135658 17837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:17:19.136019 17837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:17:19.136189 17837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:17:19.143567 17837 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:17:19.143719 17837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:17:19.143823 17837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:17:19.143926 17837 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:17:19.144014 17837 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:17:19.153997 17837 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:42195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.234.62:34075
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab
--principal=kudu/127.29.234.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:34075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.234.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:17:19.155428 17837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:17:19.156836 17837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:17:19.169209 17842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:19.169188 17843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:19.169951 17845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:17:19.169975 17837 server_base.cc:1034] running on GCE node
I20260924 16:17:19.170826 17837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:17:19.172243 17837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:17:19.173445 17837 hybrid_clock.cc:648] HybridClock initialized: now 1790266639173389 us; error 51 us; skew 500 ppm
I20260924 16:17:19.173799 17837 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:17:19 dist-test-slave-v5j9 krb5kdc[17824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266639, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:17:19.177021 17837 init.cc:377] Logged in from keytab as kudu/127.29.234.62@KRBTEST.COM (short username kudu)
I20260924 16:17:19.178534 17837 webserver.cc:460] Webserver started at http://127.29.234.62:40835/ using document root <none> and password file <none>
I20260924 16:17:19.179230 17837 fs_manager.cc:359] Metadata directory not provided
I20260924 16:17:19.179381 17837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:17:19.179698 17837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:17:19.182760 17837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/minicluster-data/master-0/data/instance:
uuid: "aa575dab39cc494392513c856da014f9"
format_stamp: "Formatted at 2026-09-24 16:17:19 on dist-test-slave-v5j9"
server_key: "38c8ad3f2491d524262c187864164b31"
server_key_iv: "7331fd407b76136967e4051b06abf68a"
server_key_version: "encryptionkey@0"
I20260924 16:17:19.183529 17837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/minicluster-data/master-0/wal/instance:
uuid: "aa575dab39cc494392513c856da014f9"
format_stamp: "Formatted at 2026-09-24 16:17:19 on dist-test-slave-v5j9"
server_key: "38c8ad3f2491d524262c187864164b31"
server_key_iv: "7331fd407b76136967e4051b06abf68a"
server_key_version: "encryptionkey@0"
I20260924 16:17:19.188759 17837 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260924 16:17:19.192437 17852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:17:19.193872 17837 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:17:19.194072 17837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/minicluster-data/master-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/minicluster-data/master-0/wal
uuid: "aa575dab39cc494392513c856da014f9"
format_stamp: "Formatted at 2026-09-24 16:17:19 on dist-test-slave-v5j9"
server_key: "38c8ad3f2491d524262c187864164b31"
server_key_iv: "7331fd407b76136967e4051b06abf68a"
server_key_version: "encryptionkey@0"
I20260924 16:17:19.194298 17837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:17:19.206373 17837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:17:19.210419 17837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:17:19.210714 17837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:17:19.229552 17837 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.62:34075
I20260924 16:17:19.229527 17904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.62:34075 every 8 connection(s)
I20260924 16:17:19.230995 17837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
I20260924 16:17:19.234925 17905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:17:19.236722 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 17837
I20260924 16:17:19.236940 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/minicluster-data/master-0/wal/instance
I20260924 16:17:19.237506 30632 external_mini_cluster.cc:1428] Setting key 12e287150ebbff0e0c0632524e3c611b
I20260924 16:17:19.248375 17905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa575dab39cc494392513c856da014f9: Bootstrap starting.
I20260924 16:17:19.252226 17905 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa575dab39cc494392513c856da014f9: Neither blocks nor log segments found. Creating new log.
I20260924 16:17:19.253288 17905 log.cc:824] T 00000000000000000000000000000000 P aa575dab39cc494392513c856da014f9: Log is configured to *not* fsync() on all Append() calls
Sep 24 16:17:19 dist-test-slave-v5j9 krb5kdc[17824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:17:19.256723 17905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa575dab39cc494392513c856da014f9: No bootstrap required, opened a new log
I20260924 16:17:19.264416 17908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:19.239887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46826 (local address 127.29.234.62:34075)
0924 16:17:19.240930 (+ 1043us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:19.240948 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:19.241325 (+ 377us) server_negotiation.cc:407] Connection header received
0924 16:17:19.243808 (+ 2483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:19.243841 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:19.244374 (+ 533us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:19.244846 (+ 472us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:19.246381 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:19.248375 (+ 1994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:19.249854 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:19.250957 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:19.251156 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:19.255355 (+ 4199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:19.255367 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:19.255390 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:17:19.255438 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:19.259532 (+ 4094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:17:19.260388 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:17:19.260403 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:17:19.260413 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:17:19.260644 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:17:19.261005 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:17:19.261009 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:17:19.261012 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:17:19.261763 (+ 751us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:19.262071 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:19.262426 (+ 355us) server_negotiation.cc:299] Negotiation successful
0924 16:17:19.262933 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":747,"thread_start_us":430,"threads_started":1}
I20260924 16:17:19.267604 17905 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa575dab39cc494392513c856da014f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa575dab39cc494392513c856da014f9" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 34075 } }
I20260924 16:17:19.267959 17905 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa575dab39cc494392513c856da014f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:17:19.268035 17905 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa575dab39cc494392513c856da014f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa575dab39cc494392513c856da014f9, State: Initialized, Role: FOLLOWER
I20260924 16:17:19.268541 17905 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa575dab39cc494392513c856da014f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa575dab39cc494392513c856da014f9" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 34075 } }
I20260924 16:17:19.268750 17905 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa575dab39cc494392513c856da014f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:17:19.268846 17905 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa575dab39cc494392513c856da014f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:17:19.269018 17905 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa575dab39cc494392513c856da014f9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:17:19.271296 17905 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa575dab39cc494392513c856da014f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa575dab39cc494392513c856da014f9" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 34075 } }
I20260924 16:17:19.271924 17905 leader_election.cc:302] T 00000000000000000000000000000000 P aa575dab39cc494392513c856da014f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa575dab39cc494392513c856da014f9; no voters:
I20260924 16:17:19.272509 17905 leader_election.cc:288] T 00000000000000000000000000000000 P aa575dab39cc494392513c856da014f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:17:19.272644 17910 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa575dab39cc494392513c856da014f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:17:19.273321 17910 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa575dab39cc494392513c856da014f9 [term 1 LEADER]: Becoming Leader. State: Replica: aa575dab39cc494392513c856da014f9, State: Running, Role: LEADER
I20260924 16:17:19.273928 17910 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa575dab39cc494392513c856da014f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa575dab39cc494392513c856da014f9" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 34075 } }
I20260924 16:17:19.274679 17905 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa575dab39cc494392513c856da014f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:17:19.276692 17911 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa575dab39cc494392513c856da014f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa575dab39cc494392513c856da014f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa575dab39cc494392513c856da014f9" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 34075 } } }
I20260924 16:17:19.277031 17911 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa575dab39cc494392513c856da014f9 [sys.catalog]: This master's current role is: LEADER
I20260924 16:17:19.277561 17912 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa575dab39cc494392513c856da014f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa575dab39cc494392513c856da014f9. Latest consensus state: current_term: 1 leader_uuid: "aa575dab39cc494392513c856da014f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa575dab39cc494392513c856da014f9" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 34075 } } }
I20260924 16:17:19.278004 17912 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa575dab39cc494392513c856da014f9 [sys.catalog]: This master's current role is: LEADER
I20260924 16:17:19.281190 17917 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:17:19.286856 17917 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:17:19.296357 17917 catalog_manager.cc:1347] Generated new cluster ID: 88ca37bab462453a91aa62d73cce535a
I20260924 16:17:19.296479 17917 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:17:19.310928 17917 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:17:19.312443 17917 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:17:19.321305 17917 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa575dab39cc494392513c856da014f9: Generated new TSK 0
I20260924 16:17:19.322193 17917 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.234.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab.
Entry for principal kudu/127.29.234.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab.
I20260924 16:17:19.385460 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.1:0
--local_ip_for_outbound_sockets=127.29.234.1
--webserver_interface=127.29.234.1
--webserver_port=0
--tserver_master_addrs=127.29.234.62:34075
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab
--principal=kudu/127.29.234.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:42195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:17:19.724848 17931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:17:19.725134 17931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:17:19.725229 17931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:17:19.732659 17931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:17:19.732769 17931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:17:19.732888 17931 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.1
I20260924 16:17:19.742727 17931 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:42195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.1.keytab
--principal=kudu/127.29.234.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.234.1
--webserver_port=0
--tserver_master_addrs=127.29.234.62:34075
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.1
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:17:19.744022 17931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:17:19.745330 17931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:17:19.759272 17939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:19.759307 17936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:19.759260 17937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:17:19.760393 17931 server_base.cc:1034] running on GCE node
I20260924 16:17:19.761118 17931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:17:19.762212 17931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:17:19.763424 17931 hybrid_clock.cc:648] HybridClock initialized: now 1790266639763385 us; error 29 us; skew 500 ppm
I20260924 16:17:19.763754 17931 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:17:19 dist-test-slave-v5j9 krb5kdc[17824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266639, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:17:19.767123 17931 init.cc:377] Logged in from keytab as kudu/127.29.234.1@KRBTEST.COM (short username kudu)
I20260924 16:17:19.768561 17931 webserver.cc:460] Webserver started at http://127.29.234.1:46739/ using document root <none> and password file <none>
I20260924 16:17:19.769259 17931 fs_manager.cc:359] Metadata directory not provided
I20260924 16:17:19.769430 17931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:17:19.769754 17931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:17:19.772655 17931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/minicluster-data/ts-0/data/instance:
uuid: "0b70a64d39ba4a13aff9c72e0e5deca2"
format_stamp: "Formatted at 2026-09-24 16:17:19 on dist-test-slave-v5j9"
server_key: "b1071c6a8e4d9e0f5369f6ac5c653d1c"
server_key_iv: "4433407f0890216c94c3644ef90ec668"
server_key_version: "encryptionkey@0"
I20260924 16:17:19.773464 17931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/minicluster-data/ts-0/wal/instance:
uuid: "0b70a64d39ba4a13aff9c72e0e5deca2"
format_stamp: "Formatted at 2026-09-24 16:17:19 on dist-test-slave-v5j9"
server_key: "b1071c6a8e4d9e0f5369f6ac5c653d1c"
server_key_iv: "4433407f0890216c94c3644ef90ec668"
server_key_version: "encryptionkey@0"
I20260924 16:17:19.778641 17931 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260924 16:17:19.782249 17946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:17:19.783680 17931 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:17:19.783851 17931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/minicluster-data/ts-0/wal
uuid: "0b70a64d39ba4a13aff9c72e0e5deca2"
format_stamp: "Formatted at 2026-09-24 16:17:19 on dist-test-slave-v5j9"
server_key: "b1071c6a8e4d9e0f5369f6ac5c653d1c"
server_key_iv: "4433407f0890216c94c3644ef90ec668"
server_key_version: "encryptionkey@0"
I20260924 16:17:19.784063 17931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:17:19.799765 17931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:17:19.804323 17931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:17:19.804641 17931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:17:19.805667 17931 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:17:19.807008 17931 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:17:19.807109 17931 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:17:19.807260 17931 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:17:19.807354 17931 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:17:19.846477 17931 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.1:34923
I20260924 16:17:19.846526 18059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.1:34923 every 8 connection(s)
I20260924 16:17:19.847815 17931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
I20260924 16:17:19.851861 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 17931
I20260924 16:17:19.852054 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/minicluster-data/ts-0/wal/instance
I20260924 16:17:19.852626 30632 external_mini_cluster.cc:1428] Setting key 9b2d3640a467b4257943dc86764f1736
Sep 24 16:17:19 dist-test-slave-v5j9 krb5kdc[17824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266639, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.1@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:17:19.872418 18062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:19.850402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.1:33335 (local address 127.29.234.62:34075)
0924 16:17:19.850840 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:19.850845 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:19.852004 (+ 1159us) server_negotiation.cc:407] Connection header received
0924 16:17:19.854417 (+ 2413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:19.854423 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:19.854527 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:19.854646 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:19.856780 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:19.857600 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:19.860059 (+ 2459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:19.860455 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:19.860555 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:19.865094 (+ 4539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:19.865104 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:19.865110 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:17:19.865157 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:19.868885 (+ 3728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:17:19.869750 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:17:19.869758 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:17:19.869764 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:17:19.869922 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:17:19.870365 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:17:19.870372 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:17:19.870378 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:17:19.870758 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:19.870935 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:19.871665 (+ 730us) server_negotiation.cc:299] Negotiation successful
0924 16:17:19.871939 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":212,"threads_started":1}
I20260924 16:17:19.873158 18063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:19.851007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:34075 (local address 127.29.234.1:33335)
0924 16:17:19.851832 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0924 16:17:19.851889 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 16:17:19.854115 (+ 2226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:17:19.854936 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:17:19.854948 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:17:19.855749 (+ 801us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:17:19.856601 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:19.856630 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:19.857775 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:19.857783 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:19.859867 (+ 2084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:19.859884 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:19.860692 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:19.860702 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:19.861066 (+ 364us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:19.861863 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:17:19.861896 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:17:19.864826 (+ 2930us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:17:19.869039 (+ 4213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:17:19.869046 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:17:19.869061 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:17:19.869598 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:17:19.870031 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:17:19.870034 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:17:19.870036 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:17:19.870245 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:17:19.870886 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:17:19.870892 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:17:19.871261 (+ 369us) client_negotiation.cc:769] Sending connection context
0924 16:17:19.871496 (+ 235us) client_negotiation.cc:240] Negotiation successful
0924 16:17:19.871902 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":268,"threads_started":1}
I20260924 16:17:19.874912 18060 heartbeater.cc:347] Connected to a master server at 127.29.234.62:34075
I20260924 16:17:19.875231 18060 heartbeater.cc:464] Registering TS with master...
I20260924 16:17:19.876051 18060 heartbeater.cc:511] Master 127.29.234.62:34075 requested a full tablet report, sending...
I20260924 16:17:19.878252 17869 ts_manager.cc:194] Registered new tserver with Master: 0b70a64d39ba4a13aff9c72e0e5deca2 (127.29.234.1:34923)
I20260924 16:17:19.879693 17869 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.234.1@KRBTEST.COM'} at 127.29.234.1:33335
WARNING: no policy specified for kudu/127.29.234.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab.
Entry for principal kudu/127.29.234.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab.
I20260924 16:17:19.913900 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.2:0
--local_ip_for_outbound_sockets=127.29.234.2
--webserver_interface=127.29.234.2
--webserver_port=0
--tserver_master_addrs=127.29.234.62:34075
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab
--principal=kudu/127.29.234.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:42195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:17:20.249539 18066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:17:20.249836 18066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:17:20.249933 18066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:17:20.257431 18066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:17:20.257594 18066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:17:20.257763 18066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.2
I20260924 16:17:20.267863 18066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:42195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.2.keytab
--principal=kudu/127.29.234.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.234.2
--webserver_port=0
--tserver_master_addrs=127.29.234.62:34075
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.2
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:17:20.269213 18066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:17:20.270642 18066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:17:20.284148 18071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:20.284138 18074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:20.284205 18072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:17:20.285074 18066 server_base.cc:1034] running on GCE node
I20260924 16:17:20.285842 18066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:17:20.286901 18066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:17:20.288185 18066 hybrid_clock.cc:648] HybridClock initialized: now 1790266640288073 us; error 61 us; skew 500 ppm
I20260924 16:17:20.288560 18066 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:17:20 dist-test-slave-v5j9 krb5kdc[17824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266640, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:17:20.291862 18066 init.cc:377] Logged in from keytab as kudu/127.29.234.2@KRBTEST.COM (short username kudu)
I20260924 16:17:20.293560 18066 webserver.cc:460] Webserver started at http://127.29.234.2:33355/ using document root <none> and password file <none>
I20260924 16:17:20.294291 18066 fs_manager.cc:359] Metadata directory not provided
I20260924 16:17:20.294442 18066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:17:20.294773 18066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:17:20.297935 18066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/minicluster-data/ts-1/data/instance:
uuid: "662e6cbd477f44efa2c7117b7092f0aa"
format_stamp: "Formatted at 2026-09-24 16:17:20 on dist-test-slave-v5j9"
server_key: "c653a1a40e9a3b84040bcc252bcc83e5"
server_key_iv: "aefab311ef63387b2bf309b96b8a445d"
server_key_version: "encryptionkey@0"
I20260924 16:17:20.298795 18066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/minicluster-data/ts-1/wal/instance:
uuid: "662e6cbd477f44efa2c7117b7092f0aa"
format_stamp: "Formatted at 2026-09-24 16:17:20 on dist-test-slave-v5j9"
server_key: "c653a1a40e9a3b84040bcc252bcc83e5"
server_key_iv: "aefab311ef63387b2bf309b96b8a445d"
server_key_version: "encryptionkey@0"
I20260924 16:17:20.304106 18066 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 16:17:20.307812 18081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:17:20.309327 18066 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 16:17:20.309587 18066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/minicluster-data/ts-1/wal
uuid: "662e6cbd477f44efa2c7117b7092f0aa"
format_stamp: "Formatted at 2026-09-24 16:17:20 on dist-test-slave-v5j9"
server_key: "c653a1a40e9a3b84040bcc252bcc83e5"
server_key_iv: "aefab311ef63387b2bf309b96b8a445d"
server_key_version: "encryptionkey@0"
I20260924 16:17:20.309805 18066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:17:20.331880 18066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:17:20.336479 18066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:17:20.336783 18066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:17:20.337927 18066 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:17:20.339372 18066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:17:20.339479 18066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:17:20.339627 18066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:17:20.339699 18066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:17:20.373652 18066 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.2:44971
I20260924 16:17:20.373759 18193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.2:44971 every 8 connection(s)
I20260924 16:17:20.375063 18066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
I20260924 16:17:20.380151 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 18066
I20260924 16:17:20.380358 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/minicluster-data/ts-1/wal/instance
I20260924 16:17:20.380924 30632 external_mini_cluster.cc:1428] Setting key ec798b8e24b011ae2e21e60f01e6a9cf
Sep 24 16:17:20 dist-test-slave-v5j9 krb5kdc[17824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266640, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.2@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:17:20.399341 18196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:20.377606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.2:42455 (local address 127.29.234.62:34075)
0924 16:17:20.378064 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:20.378069 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:20.378952 (+ 883us) server_negotiation.cc:407] Connection header received
0924 16:17:20.381445 (+ 2493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:20.381450 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:20.381527 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:20.381617 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:20.383671 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:20.384851 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:20.386533 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:20.387289 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:20.387437 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:20.391975 (+ 4538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:20.391980 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:20.391983 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:17:20.392019 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:20.395642 (+ 3623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:17:20.396527 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:17:20.396536 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:17:20.396542 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:17:20.396718 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:17:20.397136 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:17:20.397150 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:17:20.397156 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:17:20.397586 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:20.397776 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:20.398510 (+ 734us) server_negotiation.cc:299] Negotiation successful
0924 16:17:20.398778 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":246,"threads_started":1}
I20260924 16:17:20.400141 18197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:20.378100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:34075 (local address 127.29.234.2:42455)
0924 16:17:20.378793 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0924 16:17:20.378830 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 16:17:20.381091 (+ 2261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:17:20.381899 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:17:20.381908 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:17:20.382626 (+ 718us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:17:20.383466 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:20.383495 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:20.385004 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:20.385007 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:20.386366 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:20.386379 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:20.387516 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:20.387520 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:20.387724 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:20.388501 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:17:20.388537 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:17:20.391676 (+ 3139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:17:20.395870 (+ 4194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:17:20.395878 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:17:20.395893 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:17:20.396364 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:17:20.396828 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:17:20.396832 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:17:20.396834 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:17:20.397014 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:17:20.397734 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:17:20.397745 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:17:20.398125 (+ 380us) client_negotiation.cc:769] Sending connection context
0924 16:17:20.398368 (+ 243us) client_negotiation.cc:240] Negotiation successful
0924 16:17:20.398775 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":261,"threads_started":1}
I20260924 16:17:20.401767 18194 heartbeater.cc:347] Connected to a master server at 127.29.234.62:34075
I20260924 16:17:20.402127 18194 heartbeater.cc:464] Registering TS with master...
I20260924 16:17:20.403070 18194 heartbeater.cc:511] Master 127.29.234.62:34075 requested a full tablet report, sending...
I20260924 16:17:20.404778 17869 ts_manager.cc:194] Registered new tserver with Master: 662e6cbd477f44efa2c7117b7092f0aa (127.29.234.2:44971)
I20260924 16:17:20.405684 17869 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.234.2@KRBTEST.COM'} at 127.29.234.2:42455
WARNING: no policy specified for kudu/127.29.234.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab.
Entry for principal kudu/127.29.234.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab.
I20260924 16:17:20.445075 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.3:0
--local_ip_for_outbound_sockets=127.29.234.3
--webserver_interface=127.29.234.3
--webserver_port=0
--tserver_master_addrs=127.29.234.62:34075
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab
--principal=kudu/127.29.234.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:42195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:17:20.782737 18200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:17:20.783072 18200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:17:20.783238 18200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:17:20.790753 18200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:17:20.790892 18200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:17:20.791069 18200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.3
I20260924 16:17:20.800796 18200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:42195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.3.keytab
--principal=kudu/127.29.234.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.234.3
--webserver_port=0
--tserver_master_addrs=127.29.234.62:34075
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.3
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:17:20.802140 18200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:17:20.803494 18200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:17:20.816431 18205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:20.816387 18206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:20.816421 18208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:17:20.817428 18200 server_base.cc:1034] running on GCE node
I20260924 16:17:20.817998 18200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:17:20.818851 18200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:17:20.820042 18200 hybrid_clock.cc:648] HybridClock initialized: now 1790266640820004 us; error 33 us; skew 500 ppm
I20260924 16:17:20.820358 18200 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:17:20 dist-test-slave-v5j9 krb5kdc[17824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266640, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:17:20.823379 18200 init.cc:377] Logged in from keytab as kudu/127.29.234.3@KRBTEST.COM (short username kudu)
I20260924 16:17:20.824806 18200 webserver.cc:460] Webserver started at http://127.29.234.3:44955/ using document root <none> and password file <none>
I20260924 16:17:20.825518 18200 fs_manager.cc:359] Metadata directory not provided
I20260924 16:17:20.825623 18200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:17:20.825886 18200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:17:20.829253 18200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/minicluster-data/ts-2/data/instance:
uuid: "196e5a7c36dd4d69b8bfc97f5baabb0b"
format_stamp: "Formatted at 2026-09-24 16:17:20 on dist-test-slave-v5j9"
server_key: "7e86355daa96d4aa78784debb8f45373"
server_key_iv: "f0760729537b13865ea3579dec3c71c3"
server_key_version: "encryptionkey@0"
I20260924 16:17:20.830096 18200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/minicluster-data/ts-2/wal/instance:
uuid: "196e5a7c36dd4d69b8bfc97f5baabb0b"
format_stamp: "Formatted at 2026-09-24 16:17:20 on dist-test-slave-v5j9"
server_key: "7e86355daa96d4aa78784debb8f45373"
server_key_iv: "f0760729537b13865ea3579dec3c71c3"
server_key_version: "encryptionkey@0"
I20260924 16:17:20.835229 18200 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 16:17:20.839120 18216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:17:20.840422 18200 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.004s
I20260924 16:17:20.840595 18200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/minicluster-data/ts-2/wal
uuid: "196e5a7c36dd4d69b8bfc97f5baabb0b"
format_stamp: "Formatted at 2026-09-24 16:17:20 on dist-test-slave-v5j9"
server_key: "7e86355daa96d4aa78784debb8f45373"
server_key_iv: "f0760729537b13865ea3579dec3c71c3"
server_key_version: "encryptionkey@0"
I20260924 16:17:20.840798 18200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:17:20.854349 18200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:17:20.858181 18200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:17:20.858464 18200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:17:20.859395 18200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:17:20.860680 18200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:17:20.860780 18200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:17:20.860916 18200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:17:20.860991 18200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:17:20.882959 18060 heartbeater.cc:503] Master 127.29.234.62:34075 was elected leader, sending a full tablet report...
I20260924 16:17:20.894574 18200 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.3:36361
I20260924 16:17:20.894609 18330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.3:36361 every 8 connection(s)
I20260924 16:17:20.895898 18200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
I20260924 16:17:20.901880 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 18200
I20260924 16:17:20.902076 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/minicluster-data/ts-2/wal/instance
I20260924 16:17:20.902626 30632 external_mini_cluster.cc:1428] Setting key 54ac1f7780bcfe80525267c192de7959
Sep 24 16:17:20 dist-test-slave-v5j9 krb5kdc[17824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266640, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.3@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:17:20.916178 18196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:20.898571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.3:43859 (local address 127.29.234.62:34075)
0924 16:17:20.898775 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:20.898782 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:20.900026 (+ 1244us) server_negotiation.cc:407] Connection header received
0924 16:17:20.902109 (+ 2083us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:20.902114 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:20.902240 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:20.902323 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:20.904257 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:20.905373 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:20.906928 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:20.907323 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:20.907412 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:20.910837 (+ 3425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:20.910841 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:20.910845 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:17:20.910876 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:20.913289 (+ 2413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:17:20.914107 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:17:20.914110 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:17:20.914113 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:17:20.914245 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:17:20.914654 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:17:20.914657 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:17:20.914659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:17:20.914943 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:20.915054 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:20.915711 (+ 657us) server_negotiation.cc:299] Negotiation successful
0924 16:17:20.915888 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 16:17:20.917541 18333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:20.899188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:34075 (local address 127.29.234.3:43859)
0924 16:17:20.899856 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0924 16:17:20.899901 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 16:17:20.901786 (+ 1885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:17:20.902596 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:17:20.902610 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:17:20.903281 (+ 671us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:17:20.904064 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:20.904090 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:20.905527 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:20.905533 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:20.906809 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:20.906820 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:20.907502 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:20.907507 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:20.907794 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:20.908467 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:17:20.908488 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:17:20.910637 (+ 2149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:17:20.913485 (+ 2848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:17:20.913492 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:17:20.913505 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:17:20.913971 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:17:20.914373 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:17:20.914376 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:17:20.914379 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:17:20.914561 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:17:20.915042 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:17:20.915049 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:17:20.915387 (+ 338us) client_negotiation.cc:769] Sending connection context
0924 16:17:20.915620 (+ 233us) client_negotiation.cc:240] Negotiation successful
0924 16:17:20.916154 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":251,"threads_started":1}
I20260924 16:17:20.918978 18331 heartbeater.cc:347] Connected to a master server at 127.29.234.62:34075
I20260924 16:17:20.919299 18331 heartbeater.cc:464] Registering TS with master...
I20260924 16:17:20.920048 18331 heartbeater.cc:511] Master 127.29.234.62:34075 requested a full tablet report, sending...
I20260924 16:17:20.921463 17869 ts_manager.cc:194] Registered new tserver with Master: 196e5a7c36dd4d69b8bfc97f5baabb0b (127.29.234.3:36361)
I20260924 16:17:20.922256 17869 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.234.3@KRBTEST.COM'} at 127.29.234.3:43859
I20260924 16:17:20.930497 30632 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:17:20.946552 18196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:20.934633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46842 (local address 127.29.234.62:34075)
0924 16:17:20.934866 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:20.934876 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:20.935214 (+ 338us) server_negotiation.cc:407] Connection header received
0924 16:17:20.935458 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:20.935461 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:20.935532 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:20.935623 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:20.936989 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:20.937771 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:20.938953 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:20.939354 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:20.939439 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:20.941766 (+ 2327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:20.941771 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:20.941774 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:17:20.941803 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:20.944086 (+ 2283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:17:20.944705 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:17:20.944709 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:17:20.944711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:17:20.944837 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:17:20.945143 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:17:20.945147 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:17:20.945150 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:17:20.945557 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:20.945756 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:20.946028 (+ 272us) server_negotiation.cc:299] Negotiation successful
0924 16:17:20.946198 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 16:17:20.950232 17869 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46842:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:17:20.957109 17869 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:17:20.980463 18346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:20.969709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.2:44971 (local address 127.0.0.1:38608)
0924 16:17:20.971645 (+ 1936us) negotiation.cc:107] Waiting for socket to connect
0924 16:17:20.971663 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 16:17:20.971849 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:17:20.972880 (+ 1031us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:17:20.972889 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:17:20.972920 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:17:20.973291 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:20.973304 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:20.975314 (+ 2010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:20.975319 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:20.977565 (+ 2246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:20.977585 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:20.979445 (+ 1860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:20.979452 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:20.979650 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:20.979700 (+ 50us) client_negotiation.cc:769] Sending connection context
0924 16:17:20.979872 (+ 172us) client_negotiation.cc:240] Negotiation successful
0924 16:17:20.980068 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1806,"thread_start_us":1822,"threads_started":1}
I20260924 16:17:20.980491 18347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:20.969918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38608 (local address 127.29.234.2:44971)
0924 16:17:20.971223 (+ 1305us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:20.971232 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:20.971749 (+ 517us) server_negotiation.cc:407] Connection header received
0924 16:17:20.972043 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:20.972053 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:20.972357 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:20.972528 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:17:20.973499 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:20.975159 (+ 1660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:20.977759 (+ 2600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:20.979248 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:20.979402 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:20.979605 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:20.979942 (+ 337us) server_negotiation.cc:299] Negotiation successful
0924 16:17:20.980134 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1139,"thread_start_us":1788,"threads_started":1}
I20260924 16:17:20.980970 18343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:20.969441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:34923 (local address 127.0.0.1:50928)
0924 16:17:20.970339 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0924 16:17:20.970397 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 16:17:20.970601 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:17:20.973732 (+ 3131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:17:20.973736 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:17:20.973759 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:17:20.974160 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:20.974171 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:20.976135 (+ 1964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:20.976141 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:20.978229 (+ 2088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:20.978244 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:20.980112 (+ 1868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:20.980119 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:20.980302 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:20.980338 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 16:17:20.980409 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 16:17:20.980548 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"mutex_wait_us":7,"thread_start_us":671,"threads_started":1}
I20260924 16:17:20.981542 18348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:20.971919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.3:36361 (local address 127.0.0.1:33084)
0924 16:17:20.972452 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0924 16:17:20.972469 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 16:17:20.972562 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:17:20.974067 (+ 1505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:17:20.974071 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:17:20.974103 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:17:20.974877 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:20.974889 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:20.976965 (+ 2076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:20.976970 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:20.978994 (+ 2024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:20.979005 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:20.980851 (+ 1846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:20.980856 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:20.980988 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:20.981009 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 16:17:20.981109 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 16:17:20.981222 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":971,"threads_started":1}
I20260924 16:17:20.982448 18349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:20.972586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33084 (local address 127.29.234.3:36361)
0924 16:17:20.973512 (+ 926us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:20.973519 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:20.973548 (+ 29us) server_negotiation.cc:407] Connection header received
0924 16:17:20.973655 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:20.973661 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:20.973868 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:20.974058 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:17:20.975094 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:20.976803 (+ 1709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:20.979146 (+ 2343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:20.980499 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:20.980635 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:20.981776 (+ 1141us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:20.981857 (+ 81us) server_negotiation.cc:299] Negotiation successful
0924 16:17:20.982012 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":737,"thread_start_us":574,"threads_started":1}
I20260924 16:17:20.982666 18344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:20.969580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50928 (local address 127.29.234.1:34923)
0924 16:17:20.973131 (+ 3551us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:20.973139 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:20.973166 (+ 27us) server_negotiation.cc:407] Connection header received
0924 16:17:20.973269 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:20.973276 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:20.973547 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:20.973734 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:17:20.974341 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:20.975962 (+ 1621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:20.978423 (+ 2461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:20.979868 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:20.981294 (+ 1426us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:20.981544 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:20.981608 (+ 64us) server_negotiation.cc:299] Negotiation successful
0924 16:17:20.981947 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3374,"thread_start_us":3219,"threads_started":1}
I20260924 16:17:20.985140 18129 tablet_service.cc:1467] Processing CreateTablet for tablet bb6dba368cd247abbca644e6f03bd749 (DEFAULT_TABLE table=test-table [id=c5f10a90104f4fff8302fcd7884b5283]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:17:20.986483 18129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb6dba368cd247abbca644e6f03bd749. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:17:20.986290 18265 tablet_service.cc:1467] Processing CreateTablet for tablet bb6dba368cd247abbca644e6f03bd749 (DEFAULT_TABLE table=test-table [id=c5f10a90104f4fff8302fcd7884b5283]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:17:20.986534 17994 tablet_service.cc:1467] Processing CreateTablet for tablet bb6dba368cd247abbca644e6f03bd749 (DEFAULT_TABLE table=test-table [id=c5f10a90104f4fff8302fcd7884b5283]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:17:20.987896 17994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb6dba368cd247abbca644e6f03bd749. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:17:20.987926 18265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb6dba368cd247abbca644e6f03bd749. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:17:20.998601 18350 tablet_bootstrap.cc:492] T bb6dba368cd247abbca644e6f03bd749 P 662e6cbd477f44efa2c7117b7092f0aa: Bootstrap starting.
I20260924 16:17:21.002291 18351 tablet_bootstrap.cc:492] T bb6dba368cd247abbca644e6f03bd749 P 0b70a64d39ba4a13aff9c72e0e5deca2: Bootstrap starting.
I20260924 16:17:21.003139 18350 tablet_bootstrap.cc:654] T bb6dba368cd247abbca644e6f03bd749 P 662e6cbd477f44efa2c7117b7092f0aa: Neither blocks nor log segments found. Creating new log.
I20260924 16:17:21.004627 18350 log.cc:824] T bb6dba368cd247abbca644e6f03bd749 P 662e6cbd477f44efa2c7117b7092f0aa: Log is configured to *not* fsync() on all Append() calls
I20260924 16:17:21.006060 18351 tablet_bootstrap.cc:654] T bb6dba368cd247abbca644e6f03bd749 P 0b70a64d39ba4a13aff9c72e0e5deca2: Neither blocks nor log segments found. Creating new log.
I20260924 16:17:21.006383 18352 tablet_bootstrap.cc:492] T bb6dba368cd247abbca644e6f03bd749 P 196e5a7c36dd4d69b8bfc97f5baabb0b: Bootstrap starting.
I20260924 16:17:21.007339 18351 log.cc:824] T bb6dba368cd247abbca644e6f03bd749 P 0b70a64d39ba4a13aff9c72e0e5deca2: Log is configured to *not* fsync() on all Append() calls
I20260924 16:17:21.008026 18350 tablet_bootstrap.cc:492] T bb6dba368cd247abbca644e6f03bd749 P 662e6cbd477f44efa2c7117b7092f0aa: No bootstrap required, opened a new log
I20260924 16:17:21.008342 18350 ts_tablet_manager.cc:1397] T bb6dba368cd247abbca644e6f03bd749 P 662e6cbd477f44efa2c7117b7092f0aa: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260924 16:17:21.009984 18352 tablet_bootstrap.cc:654] T bb6dba368cd247abbca644e6f03bd749 P 196e5a7c36dd4d69b8bfc97f5baabb0b: Neither blocks nor log segments found. Creating new log.
I20260924 16:17:21.010730 18351 tablet_bootstrap.cc:492] T bb6dba368cd247abbca644e6f03bd749 P 0b70a64d39ba4a13aff9c72e0e5deca2: No bootstrap required, opened a new log
I20260924 16:17:21.011005 18351 ts_tablet_manager.cc:1397] T bb6dba368cd247abbca644e6f03bd749 P 0b70a64d39ba4a13aff9c72e0e5deca2: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260924 16:17:21.011365 18352 log.cc:824] T bb6dba368cd247abbca644e6f03bd749 P 196e5a7c36dd4d69b8bfc97f5baabb0b: Log is configured to *not* fsync() on all Append() calls
I20260924 16:17:21.016155 18352 tablet_bootstrap.cc:492] T bb6dba368cd247abbca644e6f03bd749 P 196e5a7c36dd4d69b8bfc97f5baabb0b: No bootstrap required, opened a new log
I20260924 16:17:21.016428 18352 ts_tablet_manager.cc:1397] T bb6dba368cd247abbca644e6f03bd749 P 196e5a7c36dd4d69b8bfc97f5baabb0b: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260924 16:17:21.017452 18350 raft_consensus.cc:348] T bb6dba368cd247abbca644e6f03bd749 P 662e6cbd477f44efa2c7117b7092f0aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b70a64d39ba4a13aff9c72e0e5deca2" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 34923 } } peers { permanent_uuid: "662e6cbd477f44efa2c7117b7092f0aa" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 44971 } } peers { permanent_uuid: "196e5a7c36dd4d69b8bfc97f5baabb0b" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 36361 } }
I20260924 16:17:21.017827 18350 raft_consensus.cc:374] T bb6dba368cd247abbca644e6f03bd749 P 662e6cbd477f44efa2c7117b7092f0aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:17:21.017951 18350 raft_consensus.cc:729] T bb6dba368cd247abbca644e6f03bd749 P 662e6cbd477f44efa2c7117b7092f0aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 662e6cbd477f44efa2c7117b7092f0aa, State: Initialized, Role: FOLLOWER
I20260924 16:17:21.018574 18350 consensus_queue.cc:260] T bb6dba368cd247abbca644e6f03bd749 P 662e6cbd477f44efa2c7117b7092f0aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b70a64d39ba4a13aff9c72e0e5deca2" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 34923 } } peers { permanent_uuid: "662e6cbd477f44efa2c7117b7092f0aa" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 44971 } } peers { permanent_uuid: "196e5a7c36dd4d69b8bfc97f5baabb0b" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 36361 } }
I20260924 16:17:21.019711 18194 heartbeater.cc:503] Master 127.29.234.62:34075 was elected leader, sending a full tablet report...
I20260924 16:17:21.020208 18350 ts_tablet_manager.cc:1428] T bb6dba368cd247abbca644e6f03bd749 P 662e6cbd477f44efa2c7117b7092f0aa: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260924 16:17:21.021903 18351 raft_consensus.cc:348] T bb6dba368cd247abbca644e6f03bd749 P 0b70a64d39ba4a13aff9c72e0e5deca2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b70a64d39ba4a13aff9c72e0e5deca2" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 34923 } } peers { permanent_uuid: "662e6cbd477f44efa2c7117b7092f0aa" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 44971 } } peers { permanent_uuid: "196e5a7c36dd4d69b8bfc97f5baabb0b" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 36361 } }
I20260924 16:17:21.022321 18351 raft_consensus.cc:374] T bb6dba368cd247abbca644e6f03bd749 P 0b70a64d39ba4a13aff9c72e0e5deca2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:17:21.022406 18351 raft_consensus.cc:729] T bb6dba368cd247abbca644e6f03bd749 P 0b70a64d39ba4a13aff9c72e0e5deca2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b70a64d39ba4a13aff9c72e0e5deca2, State: Initialized, Role: FOLLOWER
I20260924 16:17:21.022981 18351 consensus_queue.cc:260] T bb6dba368cd247abbca644e6f03bd749 P 0b70a64d39ba4a13aff9c72e0e5deca2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b70a64d39ba4a13aff9c72e0e5deca2" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 34923 } } peers { permanent_uuid: "662e6cbd477f44efa2c7117b7092f0aa" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 44971 } } peers { permanent_uuid: "196e5a7c36dd4d69b8bfc97f5baabb0b" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 36361 } }
I20260924 16:17:21.025818 18351 ts_tablet_manager.cc:1428] T bb6dba368cd247abbca644e6f03bd749 P 0b70a64d39ba4a13aff9c72e0e5deca2: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260924 16:17:21.027868 18352 raft_consensus.cc:348] T bb6dba368cd247abbca644e6f03bd749 P 196e5a7c36dd4d69b8bfc97f5baabb0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b70a64d39ba4a13aff9c72e0e5deca2" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 34923 } } peers { permanent_uuid: "662e6cbd477f44efa2c7117b7092f0aa" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 44971 } } peers { permanent_uuid: "196e5a7c36dd4d69b8bfc97f5baabb0b" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 36361 } }
I20260924 16:17:21.028230 18352 raft_consensus.cc:374] T bb6dba368cd247abbca644e6f03bd749 P 196e5a7c36dd4d69b8bfc97f5baabb0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:17:21.028344 18352 raft_consensus.cc:729] T bb6dba368cd247abbca644e6f03bd749 P 196e5a7c36dd4d69b8bfc97f5baabb0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 196e5a7c36dd4d69b8bfc97f5baabb0b, State: Initialized, Role: FOLLOWER
I20260924 16:17:21.028968 18352 consensus_queue.cc:260] T bb6dba368cd247abbca644e6f03bd749 P 196e5a7c36dd4d69b8bfc97f5baabb0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b70a64d39ba4a13aff9c72e0e5deca2" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 34923 } } peers { permanent_uuid: "662e6cbd477f44efa2c7117b7092f0aa" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 44971 } } peers { permanent_uuid: "196e5a7c36dd4d69b8bfc97f5baabb0b" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 36361 } }
I20260924 16:17:21.030085 18331 heartbeater.cc:503] Master 127.29.234.62:34075 was elected leader, sending a full tablet report...
I20260924 16:17:21.030727 18352 ts_tablet_manager.cc:1428] T bb6dba368cd247abbca644e6f03bd749 P 196e5a7c36dd4d69b8bfc97f5baabb0b: Time spent starting tablet: real 0.014s user 0.011s sys 0.003s
W20260924 16:17:21.100241 18061 tablet.cc:2369] T bb6dba368cd247abbca644e6f03bd749 P 0b70a64d39ba4a13aff9c72e0e5deca2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 16:17:21.127418 18195 tablet.cc:2369] T bb6dba368cd247abbca644e6f03bd749 P 662e6cbd477f44efa2c7117b7092f0aa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 16:17:21.147552 18332 tablet.cc:2369] T bb6dba368cd247abbca644e6f03bd749 P 196e5a7c36dd4d69b8bfc97f5baabb0b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:17:21.187606 18357 raft_consensus.cc:482] T bb6dba368cd247abbca644e6f03bd749 P 0b70a64d39ba4a13aff9c72e0e5deca2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:17:21.187896 18357 raft_consensus.cc:504] T bb6dba368cd247abbca644e6f03bd749 P 0b70a64d39ba4a13aff9c72e0e5deca2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b70a64d39ba4a13aff9c72e0e5deca2" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 34923 } } peers { permanent_uuid: "662e6cbd477f44efa2c7117b7092f0aa" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 44971 } } peers { permanent_uuid: "196e5a7c36dd4d69b8bfc97f5baabb0b" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 36361 } }
I20260924 16:17:21.189997 18357 leader_election.cc:288] T bb6dba368cd247abbca644e6f03bd749 P 0b70a64d39ba4a13aff9c72e0e5deca2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 662e6cbd477f44efa2c7117b7092f0aa (127.29.234.2:44971), 196e5a7c36dd4d69b8bfc97f5baabb0b (127.29.234.3:36361)
I20260924 16:17:21.199026 18359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:21.190479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.2:44971 (local address 127.29.234.1:43453)
0924 16:17:21.191546 (+ 1067us) negotiation.cc:107] Waiting for socket to connect
0924 16:17:21.191575 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 16:17:21.191811 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:17:21.192424 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:17:21.192429 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:17:21.192454 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:17:21.192907 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:21.192924 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:21.194475 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:21.194480 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:21.196077 (+ 1597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:21.196090 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:21.197946 (+ 1856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:21.197955 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:21.198166 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:21.198206 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 16:17:21.198304 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 16:17:21.198463 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":879,"mutex_wait_us":1,"spinlock_wait_cycles":1280,"thread_start_us":849,"threads_started":1}
I20260924 16:17:21.199049 18347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:21.190574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.1:43453 (local address 127.29.234.2:44971)
0924 16:17:21.190901 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:21.190907 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:21.191665 (+ 758us) server_negotiation.cc:407] Connection header received
0924 16:17:21.192037 (+ 372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:21.192043 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:21.192126 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:21.192212 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:17:21.193112 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:21.194337 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:21.196238 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:21.197636 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:21.197909 (+ 273us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:21.197989 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:21.198625 (+ 636us) server_negotiation.cc:299] Negotiation successful
0924 16:17:21.198778 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 16:17:21.199783 18349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:21.191134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.1:38395 (local address 127.29.234.3:36361)
0924 16:17:21.191692 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:21.191699 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:21.192293 (+ 594us) server_negotiation.cc:407] Connection header received
0924 16:17:21.192452 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:21.192458 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:21.192552 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:21.192649 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:17:21.193451 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:21.194688 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:21.197369 (+ 2681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:21.198632 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:21.198792 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:21.198859 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:21.199454 (+ 595us) server_negotiation.cc:299] Negotiation successful
0924 16:17:21.199593 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352}
I20260924 16:17:21.199880 18360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:21.191058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.3:36361 (local address 127.29.234.1:38395)
0924 16:17:21.192179 (+ 1121us) negotiation.cc:107] Waiting for socket to connect
0924 16:17:21.192195 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 16:17:21.192308 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:17:21.192703 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:17:21.192707 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:17:21.192734 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:17:21.193197 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:21.193209 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:21.194843 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:21.194854 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:21.197169 (+ 2315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:21.197187 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:21.199037 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:21.199046 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:21.199223 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:21.199253 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 16:17:21.199401 (+ 148us) client_negotiation.cc:240] Negotiation successful
0924 16:17:21.199547 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1009,"thread_start_us":757,"threads_started":1}
I20260924 16:17:21.200063 18149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb6dba368cd247abbca644e6f03bd749" candidate_uuid: "0b70a64d39ba4a13aff9c72e0e5deca2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "662e6cbd477f44efa2c7117b7092f0aa" is_pre_election: true
I20260924 16:17:21.200738 18149 raft_consensus.cc:2454] T bb6dba368cd247abbca644e6f03bd749 P 662e6cbd477f44efa2c7117b7092f0aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b70a64d39ba4a13aff9c72e0e5deca2 in term 0.
I20260924 16:17:21.200958 18285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb6dba368cd247abbca644e6f03bd749" candidate_uuid: "0b70a64d39ba4a13aff9c72e0e5deca2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "196e5a7c36dd4d69b8bfc97f5baabb0b" is_pre_election: true
I20260924 16:17:21.201519 18285 raft_consensus.cc:2454] T bb6dba368cd247abbca644e6f03bd749 P 196e5a7c36dd4d69b8bfc97f5baabb0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b70a64d39ba4a13aff9c72e0e5deca2 in term 0.
I20260924 16:17:21.201508 17950 leader_election.cc:302] T bb6dba368cd247abbca644e6f03bd749 P 0b70a64d39ba4a13aff9c72e0e5deca2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b70a64d39ba4a13aff9c72e0e5deca2, 662e6cbd477f44efa2c7117b7092f0aa; no voters:
I20260924 16:17:21.201896 18357 raft_consensus.cc:2781] T bb6dba368cd247abbca644e6f03bd749 P 0b70a64d39ba4a13aff9c72e0e5deca2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:17:21.202081 18357 raft_consensus.cc:482] T bb6dba368cd247abbca644e6f03bd749 P 0b70a64d39ba4a13aff9c72e0e5deca2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:17:21.202225 18357 raft_consensus.cc:3037] T bb6dba368cd247abbca644e6f03bd749 P 0b70a64d39ba4a13aff9c72e0e5deca2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:17:21.204399 18357 raft_consensus.cc:504] T bb6dba368cd247abbca644e6f03bd749 P 0b70a64d39ba4a13aff9c72e0e5deca2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b70a64d39ba4a13aff9c72e0e5deca2" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 34923 } } peers { permanent_uuid: "662e6cbd477f44efa2c7117b7092f0aa" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 44971 } } peers { permanent_uuid: "196e5a7c36dd4d69b8bfc97f5baabb0b" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 36361 } }
I20260924 16:17:21.205046 18357 leader_election.cc:288] T bb6dba368cd247abbca644e6f03bd749 P 0b70a64d39ba4a13aff9c72e0e5deca2 [CANDIDATE]: Term 1 election: Requested vote from peers 662e6cbd477f44efa2c7117b7092f0aa (127.29.234.2:44971), 196e5a7c36dd4d69b8bfc97f5baabb0b (127.29.234.3:36361)
I20260924 16:17:21.205435 18149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb6dba368cd247abbca644e6f03bd749" candidate_uuid: "0b70a64d39ba4a13aff9c72e0e5deca2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "662e6cbd477f44efa2c7117b7092f0aa"
I20260924 16:17:21.205466 18285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb6dba368cd247abbca644e6f03bd749" candidate_uuid: "0b70a64d39ba4a13aff9c72e0e5deca2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "196e5a7c36dd4d69b8bfc97f5baabb0b"
I20260924 16:17:21.205641 18149 raft_consensus.cc:3037] T bb6dba368cd247abbca644e6f03bd749 P 662e6cbd477f44efa2c7117b7092f0aa [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:17:21.205641 18285 raft_consensus.cc:3037] T bb6dba368cd247abbca644e6f03bd749 P 196e5a7c36dd4d69b8bfc97f5baabb0b [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:17:21.208844 18149 raft_consensus.cc:2454] T bb6dba368cd247abbca644e6f03bd749 P 662e6cbd477f44efa2c7117b7092f0aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b70a64d39ba4a13aff9c72e0e5deca2 in term 1.
I20260924 16:17:21.208839 18285 raft_consensus.cc:2454] T bb6dba368cd247abbca644e6f03bd749 P 196e5a7c36dd4d69b8bfc97f5baabb0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b70a64d39ba4a13aff9c72e0e5deca2 in term 1.
I20260924 16:17:21.209260 17949 leader_election.cc:302] T bb6dba368cd247abbca644e6f03bd749 P 0b70a64d39ba4a13aff9c72e0e5deca2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b70a64d39ba4a13aff9c72e0e5deca2, 196e5a7c36dd4d69b8bfc97f5baabb0b; no voters:
I20260924 16:17:21.209623 18357 raft_consensus.cc:2781] T bb6dba368cd247abbca644e6f03bd749 P 0b70a64d39ba4a13aff9c72e0e5deca2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:17:21.210057 18357 raft_consensus.cc:686] T bb6dba368cd247abbca644e6f03bd749 P 0b70a64d39ba4a13aff9c72e0e5deca2 [term 1 LEADER]: Becoming Leader. State: Replica: 0b70a64d39ba4a13aff9c72e0e5deca2, State: Running, Role: LEADER
I20260924 16:17:21.210566 18357 consensus_queue.cc:237] T bb6dba368cd247abbca644e6f03bd749 P 0b70a64d39ba4a13aff9c72e0e5deca2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b70a64d39ba4a13aff9c72e0e5deca2" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 34923 } } peers { permanent_uuid: "662e6cbd477f44efa2c7117b7092f0aa" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 44971 } } peers { permanent_uuid: "196e5a7c36dd4d69b8bfc97f5baabb0b" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 36361 } }
I20260924 16:17:21.215866 17868 catalog_manager.cc:5495] T bb6dba368cd247abbca644e6f03bd749 P 0b70a64d39ba4a13aff9c72e0e5deca2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b70a64d39ba4a13aff9c72e0e5deca2 (127.29.234.1). New cstate: current_term: 1 leader_uuid: "0b70a64d39ba4a13aff9c72e0e5deca2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b70a64d39ba4a13aff9c72e0e5deca2" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 34923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "662e6cbd477f44efa2c7117b7092f0aa" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 44971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "196e5a7c36dd4d69b8bfc97f5baabb0b" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 36361 } health_report { overall_health: UNKNOWN } } }
I20260924 16:17:21.281335 18344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:21.276121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50932 (local address 127.29.234.1:34923)
0924 16:17:21.276326 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:21.276331 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:21.276349 (+ 18us) server_negotiation.cc:407] Connection header received
0924 16:17:21.276544 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:21.276548 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:21.276644 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:21.276770 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:17:21.277097 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:21.277869 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:21.279192 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:21.279596 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:21.279694 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:21.280264 (+ 570us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:17:21.280831 (+ 567us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:17:21.280942 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:21.281041 (+ 99us) server_negotiation.cc:299] Negotiation successful
0924 16:17:21.281109 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 16:17:21.289861 18285 raft_consensus.cc:1261] T bb6dba368cd247abbca644e6f03bd749 P 196e5a7c36dd4d69b8bfc97f5baabb0b [term 1 FOLLOWER]: Refusing update from remote peer 0b70a64d39ba4a13aff9c72e0e5deca2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:17:21.289861 18149 raft_consensus.cc:1261] T bb6dba368cd247abbca644e6f03bd749 P 662e6cbd477f44efa2c7117b7092f0aa [term 1 FOLLOWER]: Refusing update from remote peer 0b70a64d39ba4a13aff9c72e0e5deca2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:17:21.290963 18357 consensus_queue.cc:1035] T bb6dba368cd247abbca644e6f03bd749 P 0b70a64d39ba4a13aff9c72e0e5deca2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "662e6cbd477f44efa2c7117b7092f0aa" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 44971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:17:21.291276 18361 consensus_queue.cc:1035] T bb6dba368cd247abbca644e6f03bd749 P 0b70a64d39ba4a13aff9c72e0e5deca2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "196e5a7c36dd4d69b8bfc97f5baabb0b" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 36361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:17:21.303704 18364 mvcc.cc:205] Tried to move back new op lower bound from 7332932162712526848 to 7332932162403270656. Current Snapshot: MvccSnapshot[applied={T|T < 7332932162712526848}]
I20260924 16:17:21.307075 18365 mvcc.cc:205] Tried to move back new op lower bound from 7332932162712526848 to 7332932162403270656. Current Snapshot: MvccSnapshot[applied={T|T < 7332932162712526848}]
I20260924 16:17:21.317970 17868 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46842:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:17:21.318241 17868 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46842:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:17:21.321867 17868 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aa575dab39cc494392513c856da014f9: Sending DeleteTablet for 3 replicas of tablet bb6dba368cd247abbca644e6f03bd749
I20260924 16:17:21.323195 18129 tablet_service.cc:1514] Processing DeleteTablet for tablet bb6dba368cd247abbca644e6f03bd749 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:17:21 UTC) from {username='kudu', principal='kudu/127.29.234.62@KRBTEST.COM'} at 127.0.0.1:38608
I20260924 16:17:21.323220 18265 tablet_service.cc:1514] Processing DeleteTablet for tablet bb6dba368cd247abbca644e6f03bd749 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:17:21 UTC) from {username='kudu', principal='kudu/127.29.234.62@KRBTEST.COM'} at 127.0.0.1:33084
I20260924 16:17:21.323937 17994 tablet_service.cc:1514] Processing DeleteTablet for tablet bb6dba368cd247abbca644e6f03bd749 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:17:21 UTC) from {username='kudu', principal='kudu/127.29.234.62@KRBTEST.COM'} at 127.0.0.1:50928
I20260924 16:17:21.324792 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 17931
I20260924 16:17:21.334820 18374 tablet_replica.cc:330] T bb6dba368cd247abbca644e6f03bd749 P 196e5a7c36dd4d69b8bfc97f5baabb0b: stopping tablet replica
I20260924 16:17:21.336720 18374 raft_consensus.cc:2229] T bb6dba368cd247abbca644e6f03bd749 P 196e5a7c36dd4d69b8bfc97f5baabb0b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:17:21.338748 18374 raft_consensus.cc:2258] T bb6dba368cd247abbca644e6f03bd749 P 196e5a7c36dd4d69b8bfc97f5baabb0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:17:21.339700 18375 tablet_replica.cc:330] T bb6dba368cd247abbca644e6f03bd749 P 0b70a64d39ba4a13aff9c72e0e5deca2: stopping tablet replica
I20260924 16:17:21.339988 18373 tablet_replica.cc:330] T bb6dba368cd247abbca644e6f03bd749 P 662e6cbd477f44efa2c7117b7092f0aa: stopping tablet replica
I20260924 16:17:21.340173 18375 raft_consensus.cc:2229] T bb6dba368cd247abbca644e6f03bd749 P 0b70a64d39ba4a13aff9c72e0e5deca2 [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:17:21.340469 18373 raft_consensus.cc:2229] T bb6dba368cd247abbca644e6f03bd749 P 662e6cbd477f44efa2c7117b7092f0aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:17:21.340880 18375 raft_consensus.cc:2258] T bb6dba368cd247abbca644e6f03bd749 P 0b70a64d39ba4a13aff9c72e0e5deca2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:17:21.341043 18373 raft_consensus.cc:2258] T bb6dba368cd247abbca644e6f03bd749 P 662e6cbd477f44efa2c7117b7092f0aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:17:21.341985 18374 ts_tablet_manager.cc:1905] T bb6dba368cd247abbca644e6f03bd749 P 196e5a7c36dd4d69b8bfc97f5baabb0b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:17:21.343326 18375 ts_tablet_manager.cc:1905] T bb6dba368cd247abbca644e6f03bd749 P 0b70a64d39ba4a13aff9c72e0e5deca2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:17:21.343326 18373 ts_tablet_manager.cc:1905] T bb6dba368cd247abbca644e6f03bd749 P 662e6cbd477f44efa2c7117b7092f0aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:17:21.350034 18374 ts_tablet_manager.cc:1918] T bb6dba368cd247abbca644e6f03bd749 P 196e5a7c36dd4d69b8bfc97f5baabb0b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:17:21.350236 18374 log.cc:1196] T bb6dba368cd247abbca644e6f03bd749 P 196e5a7c36dd4d69b8bfc97f5baabb0b: Deleting WAL directory at /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/minicluster-data/ts-2/wal/wals/bb6dba368cd247abbca644e6f03bd749
I20260924 16:17:21.350862 18374 ts_tablet_manager.cc:1939] T bb6dba368cd247abbca644e6f03bd749 P 196e5a7c36dd4d69b8bfc97f5baabb0b: Deleting consensus metadata
I20260924 16:17:21.351107 18375 ts_tablet_manager.cc:1918] T bb6dba368cd247abbca644e6f03bd749 P 0b70a64d39ba4a13aff9c72e0e5deca2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:17:21.351311 18375 log.cc:1196] T bb6dba368cd247abbca644e6f03bd749 P 0b70a64d39ba4a13aff9c72e0e5deca2: Deleting WAL directory at /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/minicluster-data/ts-0/wal/wals/bb6dba368cd247abbca644e6f03bd749
I20260924 16:17:21.351910 18375 ts_tablet_manager.cc:1939] T bb6dba368cd247abbca644e6f03bd749 P 0b70a64d39ba4a13aff9c72e0e5deca2: Deleting consensus metadata
Sep 24 16:17:21 dist-test-slave-v5j9 krb5kdc[17824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.234.1@KRBTEST.COM
I20260924 16:17:21.353108 17855 catalog_manager.cc:4841] TS 196e5a7c36dd4d69b8bfc97f5baabb0b (127.29.234.3:36361): tablet bb6dba368cd247abbca644e6f03bd749 (table test-table [id=c5f10a90104f4fff8302fcd7884b5283]) successfully deleted
I20260924 16:17:21.353808 17855 catalog_manager.cc:4841] TS 0b70a64d39ba4a13aff9c72e0e5deca2 (127.29.234.1:34923): tablet bb6dba368cd247abbca644e6f03bd749 (table test-table [id=c5f10a90104f4fff8302fcd7884b5283]) successfully deleted
I20260924 16:17:21.355881 18373 ts_tablet_manager.cc:1918] T bb6dba368cd247abbca644e6f03bd749 P 662e6cbd477f44efa2c7117b7092f0aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:17:21.356050 18373 log.cc:1196] T bb6dba368cd247abbca644e6f03bd749 P 662e6cbd477f44efa2c7117b7092f0aa: Deleting WAL directory at /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790266323169505-30632-0/minicluster-data/ts-1/wal/wals/bb6dba368cd247abbca644e6f03bd749
I20260924 16:17:21.356613 18373 ts_tablet_manager.cc:1939] T bb6dba368cd247abbca644e6f03bd749 P 662e6cbd477f44efa2c7117b7092f0aa: Deleting consensus metadata
I20260924 16:17:21.358075 17856 catalog_manager.cc:4841] TS 662e6cbd477f44efa2c7117b7092f0aa (127.29.234.2:44971): tablet bb6dba368cd247abbca644e6f03bd749 (table test-table [id=c5f10a90104f4fff8302fcd7884b5283]) successfully deleted
I20260924 16:17:21.361191 18344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:21.325768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50944 (local address 127.29.234.1:34923)
0924 16:17:21.325976 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:21.325982 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:21.336300 (+ 10318us) server_negotiation.cc:407] Connection header received
0924 16:17:21.336880 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:21.336887 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:21.336985 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:21.337082 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:21.341281 (+ 4199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:21.342488 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:21.344453 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:21.345191 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:21.345296 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:21.354003 (+ 8707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:21.354018 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:21.354038 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:17:21.354092 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:21.358274 (+ 4182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:17:21.358940 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:17:21.358950 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:17:21.358954 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:17:21.359138 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:17:21.359538 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:17:21.359543 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:17:21.359547 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:17:21.359996 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:21.360325 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:21.360536 (+ 211us) server_negotiation.cc:299] Negotiation successful
0924 16:17:21.360789 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 16:17:21.361946 18055 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:17:21.460603 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 17931
I20260924 16:17:21.480798 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 18066
Sep 24 16:17:21 dist-test-slave-v5j9 krb5kdc[17824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.234.2@KRBTEST.COM
I20260924 16:17:21.495957 18347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:21.481721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38612 (local address 127.29.234.2:44971)
0924 16:17:21.481916 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:21.481921 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:21.481938 (+ 17us) server_negotiation.cc:407] Connection header received
0924 16:17:21.482140 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:21.482145 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:21.482224 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:21.482327 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:21.483707 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:21.484454 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:21.485716 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:21.486095 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:21.486192 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:21.490312 (+ 4120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:21.490325 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:21.490342 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:17:21.490371 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:21.492980 (+ 2609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:17:21.493825 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:17:21.493832 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:17:21.493834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:17:21.493952 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:17:21.494348 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:17:21.494353 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:17:21.494357 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:17:21.494733 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:21.495081 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:21.495391 (+ 310us) server_negotiation.cc:299] Negotiation successful
0924 16:17:21.495601 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 16:17:21.496626 18189 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:17:21.587041 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 18066
I20260924 16:17:21.605800 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 18200
Sep 24 16:17:21 dist-test-slave-v5j9 krb5kdc[17824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.234.3@KRBTEST.COM
I20260924 16:17:21.621297 18349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:21.606633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33092 (local address 127.29.234.3:36361)
0924 16:17:21.606827 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:21.606835 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:21.606895 (+ 60us) server_negotiation.cc:407] Connection header received
0924 16:17:21.607127 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:21.607131 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:21.607212 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:21.607326 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:21.608752 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:21.609542 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:21.610851 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:21.611251 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:21.611383 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:21.615988 (+ 4605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:21.616004 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:21.616026 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:17:21.616061 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:21.618832 (+ 2771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:17:21.619512 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:17:21.619521 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:17:21.619523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:17:21.619691 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:17:21.620015 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:17:21.620018 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:17:21.620020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:17:21.620389 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:21.620622 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:21.620848 (+ 226us) server_negotiation.cc:299] Negotiation successful
0924 16:17:21.621014 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 16:17:21.622013 18326 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:17:21.710426 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 18200
I20260924 16:17:21.728078 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 17837
I20260924 16:17:21.729210 17900 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:17:21.842412 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 17837
2026-09-24T16:17:21Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4424 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-24T16:17:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:17:21Z Disabled control of system clock
I20260924 16:17:21.923727 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266323169505-30632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266323169505-30632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266323169505-30632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:46225
--webserver_interface=127.29.234.62
--webserver_port=0
--builtin_ntp_servers=127.29.234.20:45599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.234.62:46225
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 16:17:22.279273 18384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:17:22.279592 18384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:17:22.279759 18384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:17:22.287112 18384 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:17:22.287282 18384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:17:22.287340 18384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:17:22.287398 18384 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:17:22.287442 18384 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:17:22.297464 18384 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:45599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266323169505-30632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266323169505-30632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.234.62:46225
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:46225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.234.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266323169505-30632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:17:22.298840 18384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:17:22.300561 18384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:17:22.312271 18389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:22.312312 18393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:22.312448 18390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:17:22.313495 18384 server_base.cc:1034] running on GCE node
I20260924 16:17:22.314268 18384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:17:22.315711 18384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:17:22.316939 18384 hybrid_clock.cc:648] HybridClock initialized: now 1790266642316893 us; error 40 us; skew 500 ppm
I20260924 16:17:22.317305 18384 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 16:17:22.325451 18384 webserver.cc:460] Webserver started at http://127.29.234.62:44127/ using document root <none> and password file <none>
I20260924 16:17:22.326258 18384 fs_manager.cc:359] Metadata directory not provided
I20260924 16:17:22.326398 18384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:17:22.326740 18384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:17:22.329869 18384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266323169505-30632-0/minicluster-data/master-0/data/instance:
uuid: "1edc27698e484944b1f945e20140925a"
format_stamp: "Formatted at 2026-09-24 16:17:22 on dist-test-slave-v5j9"
server_key: "934d64557165086133883768f527a404"
server_key_iv: "e1c7a85b584919683a40e37e7fd17ca5"
server_key_version: "encryptionkey@0"
I20260924 16:17:22.330698 18384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266323169505-30632-0/minicluster-data/master-0/wal/instance:
uuid: "1edc27698e484944b1f945e20140925a"
format_stamp: "Formatted at 2026-09-24 16:17:22 on dist-test-slave-v5j9"
server_key: "934d64557165086133883768f527a404"
server_key_iv: "e1c7a85b584919683a40e37e7fd17ca5"
server_key_version: "encryptionkey@0"
I20260924 16:17:22.336079 18384 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 16:17:22.339807 18399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:17:22.341281 18384 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 16:17:22.341504 18384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266323169505-30632-0/minicluster-data/master-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266323169505-30632-0/minicluster-data/master-0/wal
uuid: "1edc27698e484944b1f945e20140925a"
format_stamp: "Formatted at 2026-09-24 16:17:22 on dist-test-slave-v5j9"
server_key: "934d64557165086133883768f527a404"
server_key_iv: "e1c7a85b584919683a40e37e7fd17ca5"
server_key_version: "encryptionkey@0"
I20260924 16:17:22.341689 18384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266323169505-30632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266323169505-30632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266323169505-30632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:17:22.358844 18384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:17:22.359722 18384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:17:22.359969 18384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:17:22.378714 18384 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.62:46225
I20260924 16:17:22.378734 18451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.62:46225 every 8 connection(s)
I20260924 16:17:22.380101 18384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
I20260924 16:17:22.383899 18452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:17:22.390475 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 18384
I20260924 16:17:22.390676 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266323169505-30632-0/minicluster-data/master-0/wal/instance
I20260924 16:17:22.391265 30632 external_mini_cluster.cc:1428] Setting key b9674e7f5b4f224b19a21d42df0d8e2e
I20260924 16:17:22.395401 18452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1edc27698e484944b1f945e20140925a: Bootstrap starting.
I20260924 16:17:22.398643 18452 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1edc27698e484944b1f945e20140925a: Neither blocks nor log segments found. Creating new log.
I20260924 16:17:22.399667 18452 log.cc:824] T 00000000000000000000000000000000 P 1edc27698e484944b1f945e20140925a: Log is configured to *not* fsync() on all Append() calls
I20260924 16:17:22.403015 18452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1edc27698e484944b1f945e20140925a: No bootstrap required, opened a new log
I20260924 16:17:22.405061 18455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:22.393247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59024 (local address 127.29.234.62:46225)
0924 16:17:22.394276 (+ 1029us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:22.394290 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:22.394334 (+ 44us) server_negotiation.cc:407] Connection header received
0924 16:17:22.396110 (+ 1776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:22.396135 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:22.396594 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:22.397007 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:22.397612 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:22.398938 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:22.400532 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:22.401258 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:22.401461 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:22.402681 (+ 1220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:22.402690 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:22.402705 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:17:22.403289 (+ 584us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:22.403367 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:17:22.403392 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:22.403485 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:22.403754 (+ 269us) server_negotiation.cc:299] Negotiation successful
0924 16:17:22.404097 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":332,"threads_started":1}
I20260924 16:17:22.411319 18452 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1edc27698e484944b1f945e20140925a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1edc27698e484944b1f945e20140925a" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 46225 } }
I20260924 16:17:22.411671 18452 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1edc27698e484944b1f945e20140925a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:17:22.411795 18452 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1edc27698e484944b1f945e20140925a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1edc27698e484944b1f945e20140925a, State: Initialized, Role: FOLLOWER
I20260924 16:17:22.412356 18452 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1edc27698e484944b1f945e20140925a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1edc27698e484944b1f945e20140925a" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 46225 } }
I20260924 16:17:22.412585 18452 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1edc27698e484944b1f945e20140925a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:17:22.412719 18452 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1edc27698e484944b1f945e20140925a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:17:22.412892 18452 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1edc27698e484944b1f945e20140925a [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:17:22.415300 18452 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1edc27698e484944b1f945e20140925a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1edc27698e484944b1f945e20140925a" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 46225 } }
I20260924 16:17:22.415783 18452 leader_election.cc:302] T 00000000000000000000000000000000 P 1edc27698e484944b1f945e20140925a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1edc27698e484944b1f945e20140925a; no voters:
I20260924 16:17:22.416383 18452 leader_election.cc:288] T 00000000000000000000000000000000 P 1edc27698e484944b1f945e20140925a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:17:22.416566 18457 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1edc27698e484944b1f945e20140925a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:17:22.417309 18457 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1edc27698e484944b1f945e20140925a [term 1 LEADER]: Becoming Leader. State: Replica: 1edc27698e484944b1f945e20140925a, State: Running, Role: LEADER
I20260924 16:17:22.417871 18457 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1edc27698e484944b1f945e20140925a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1edc27698e484944b1f945e20140925a" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 46225 } }
I20260924 16:17:22.418489 18452 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1edc27698e484944b1f945e20140925a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:17:22.422256 18459 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1edc27698e484944b1f945e20140925a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1edc27698e484944b1f945e20140925a. Latest consensus state: current_term: 1 leader_uuid: "1edc27698e484944b1f945e20140925a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1edc27698e484944b1f945e20140925a" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 46225 } } }
I20260924 16:17:22.422192 18458 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1edc27698e484944b1f945e20140925a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1edc27698e484944b1f945e20140925a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1edc27698e484944b1f945e20140925a" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 46225 } } }
I20260924 16:17:22.422680 18459 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1edc27698e484944b1f945e20140925a [sys.catalog]: This master's current role is: LEADER
I20260924 16:17:22.422770 18458 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1edc27698e484944b1f945e20140925a [sys.catalog]: This master's current role is: LEADER
I20260924 16:17:22.424778 18465 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:17:22.430365 18465 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:17:22.439575 18465 catalog_manager.cc:1347] Generated new cluster ID: c9f64a8d132d42ed917a6b19014a4439
I20260924 16:17:22.439703 18465 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:17:22.453668 18465 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:17:22.455535 18465 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:17:22.463220 18465 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1edc27698e484944b1f945e20140925a: Generated new TSK 0
I20260924 16:17:22.464284 18465 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:17:22.476305 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266323169505-30632-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266323169505-30632-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266323169505-30632-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.1:0
--local_ip_for_outbound_sockets=127.29.234.1
--webserver_interface=127.29.234.1
--webserver_port=0
--tserver_master_addrs=127.29.234.62:46225
--builtin_ntp_servers=127.29.234.20:45599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 16:17:22.812986 18476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:17:22.813246 18476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:17:22.813407 18476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:17:22.820606 18476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:17:22.820705 18476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:17:22.820906 18476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.1
I20260924 16:17:22.830097 18476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:45599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266323169505-30632-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266323169505-30632-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.234.1
--webserver_port=0
--tserver_master_addrs=127.29.234.62:46225
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.1
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266323169505-30632-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:17:22.831384 18476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:17:22.833462 18476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:17:22.846689 18481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:22.846711 18482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:22.846765 18484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:17:22.847527 18476 server_base.cc:1034] running on GCE node
I20260924 16:17:22.848119 18476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:17:22.849100 18476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:17:22.850342 18476 hybrid_clock.cc:648] HybridClock initialized: now 1790266642850266 us; error 66 us; skew 500 ppm
I20260924 16:17:22.850666 18476 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 16:17:22.852819 18476 webserver.cc:460] Webserver started at http://127.29.234.1:45123/ using document root <none> and password file <none>
I20260924 16:17:22.853607 18476 fs_manager.cc:359] Metadata directory not provided
I20260924 16:17:22.853753 18476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:17:22.854063 18476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:17:22.856861 18476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266323169505-30632-0/minicluster-data/ts-0/data/instance:
uuid: "ff468e18221242c9ab8974e5eef0c229"
format_stamp: "Formatted at 2026-09-24 16:17:22 on dist-test-slave-v5j9"
server_key: "8236a7efa3cf25937d7adcc1606fb095"
server_key_iv: "56046b6cb18bcb50f8f4df02a089b9a3"
server_key_version: "encryptionkey@0"
I20260924 16:17:22.857714 18476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266323169505-30632-0/minicluster-data/ts-0/wal/instance:
uuid: "ff468e18221242c9ab8974e5eef0c229"
format_stamp: "Formatted at 2026-09-24 16:17:22 on dist-test-slave-v5j9"
server_key: "8236a7efa3cf25937d7adcc1606fb095"
server_key_iv: "56046b6cb18bcb50f8f4df02a089b9a3"
server_key_version: "encryptionkey@0"
I20260924 16:17:22.862710 18476 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260924 16:17:22.866186 18490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:17:22.867511 18476 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260924 16:17:22.867692 18476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266323169505-30632-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266323169505-30632-0/minicluster-data/ts-0/wal
uuid: "ff468e18221242c9ab8974e5eef0c229"
format_stamp: "Formatted at 2026-09-24 16:17:22 on dist-test-slave-v5j9"
server_key: "8236a7efa3cf25937d7adcc1606fb095"
server_key_iv: "56046b6cb18bcb50f8f4df02a089b9a3"
server_key_version: "encryptionkey@0"
I20260924 16:17:22.867911 18476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266323169505-30632-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266323169505-30632-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266323169505-30632-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:17:22.892930 18476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:17:22.893960 18476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:17:22.894310 18476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:17:22.895368 18476 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:17:22.896874 18476 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:17:22.896997 18476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:17:22.897135 18476 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:17:22.897254 18476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:17:22.931906 18476 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.1:41533
I20260924 16:17:22.931972 18604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.1:41533 every 8 connection(s)
I20260924 16:17:22.933310 18476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266323169505-30632-0/minicluster-data/ts-0/data/info.pb
I20260924 16:17:22.934912 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 18476
I20260924 16:17:22.935108 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266323169505-30632-0/minicluster-data/ts-0/wal/instance
I20260924 16:17:22.935607 30632 external_mini_cluster.cc:1428] Setting key a81c8dc589e50fb95750f6eb4a459abf
I20260924 16:17:22.938634 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266323169505-30632-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266323169505-30632-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266323169505-30632-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.2:0
--local_ip_for_outbound_sockets=127.29.234.2
--webserver_interface=127.29.234.2
--webserver_port=0
--tserver_master_addrs=127.29.234.62:46225
--builtin_ntp_servers=127.29.234.20:45599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 16:17:22.948096 18607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:22.936285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.1:56075 (local address 127.29.234.62:46225)
0924 16:17:22.936762 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:22.936768 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:22.938373 (+ 1605us) server_negotiation.cc:407] Connection header received
0924 16:17:22.940392 (+ 2019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:22.940401 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:22.940500 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:22.940624 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:22.941845 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:22.943040 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:22.944629 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:22.945414 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:22.945557 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:22.947259 (+ 1702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:22.947263 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:22.947267 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:17:22.947298 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:22.947341 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:17:22.947353 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:22.947421 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:22.947727 (+ 306us) server_negotiation.cc:299] Negotiation successful
0924 16:17:22.947818 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":240,"threads_started":1}
I20260924 16:17:22.949229 18608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:22.937055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:46225 (local address 127.29.234.1:56075)
0924 16:17:22.938190 (+ 1135us) negotiation.cc:107] Waiting for socket to connect
0924 16:17:22.938236 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 16:17:22.940100 (+ 1864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:17:22.940838 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:17:22.940847 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:17:22.940948 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:17:22.941665 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:22.941686 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:22.943214 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:22.943218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:22.944476 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:22.944489 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:22.945670 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:22.945676 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:22.945963 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:22.946812 (+ 849us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:17:22.946818 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:17:22.946883 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:17:22.947007 (+ 124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:17:22.947451 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:17:22.947463 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:17:22.947472 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 16:17:22.947672 (+ 200us) client_negotiation.cc:240] Negotiation successful
0924 16:17:22.948131 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":360,"threads_started":1}
I20260924 16:17:22.951040 18605 heartbeater.cc:347] Connected to a master server at 127.29.234.62:46225
I20260924 16:17:22.951363 18605 heartbeater.cc:464] Registering TS with master...
I20260924 16:17:22.952123 18605 heartbeater.cc:511] Master 127.29.234.62:46225 requested a full tablet report, sending...
I20260924 16:17:22.954245 18416 ts_manager.cc:194] Registered new tserver with Master: ff468e18221242c9ab8974e5eef0c229 (127.29.234.1:41533)
I20260924 16:17:22.955745 18416 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.234.1:56075
W20260924 16:17:23.278756 18609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:17:23.279098 18609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:17:23.279260 18609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:17:23.286674 18609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:17:23.286804 18609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:17:23.286976 18609 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.2
I20260924 16:17:23.296211 18609 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:45599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266323169505-30632-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266323169505-30632-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.234.2
--webserver_port=0
--tserver_master_addrs=127.29.234.62:46225
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.2
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266323169505-30632-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:17:23.297712 18609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:17:23.299988 18609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:17:23.313658 18614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:23.313611 18615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:23.313692 18617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:17:23.314510 18609 server_base.cc:1034] running on GCE node
I20260924 16:17:23.315173 18609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:17:23.316164 18609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:17:23.317456 18609 hybrid_clock.cc:648] HybridClock initialized: now 1790266643317407 us; error 30 us; skew 500 ppm
I20260924 16:17:23.317816 18609 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 16:17:23.320286 18609 webserver.cc:460] Webserver started at http://127.29.234.2:43793/ using document root <none> and password file <none>
I20260924 16:17:23.321003 18609 fs_manager.cc:359] Metadata directory not provided
I20260924 16:17:23.321141 18609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:17:23.321549 18609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:17:23.324432 18609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266323169505-30632-0/minicluster-data/ts-1/data/instance:
uuid: "6c037edff4604753ab024af3eb260a99"
format_stamp: "Formatted at 2026-09-24 16:17:23 on dist-test-slave-v5j9"
server_key: "f8154a6a419ed1c011a819dd0d3f287e"
server_key_iv: "627906997cb4e6637a7e617ae1734ee7"
server_key_version: "encryptionkey@0"
I20260924 16:17:23.325403 18609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266323169505-30632-0/minicluster-data/ts-1/wal/instance:
uuid: "6c037edff4604753ab024af3eb260a99"
format_stamp: "Formatted at 2026-09-24 16:17:23 on dist-test-slave-v5j9"
server_key: "f8154a6a419ed1c011a819dd0d3f287e"
server_key_iv: "627906997cb4e6637a7e617ae1734ee7"
server_key_version: "encryptionkey@0"
I20260924 16:17:23.330616 18609 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 16:17:23.334141 18623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:17:23.335596 18609 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260924 16:17:23.335795 18609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266323169505-30632-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266323169505-30632-0/minicluster-data/ts-1/wal
uuid: "6c037edff4604753ab024af3eb260a99"
format_stamp: "Formatted at 2026-09-24 16:17:23 on dist-test-slave-v5j9"
server_key: "f8154a6a419ed1c011a819dd0d3f287e"
server_key_iv: "627906997cb4e6637a7e617ae1734ee7"
server_key_version: "encryptionkey@0"
I20260924 16:17:23.336030 18609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266323169505-30632-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266323169505-30632-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266323169505-30632-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:17:23.353171 18609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:17:23.354122 18609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:17:23.354452 18609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:17:23.355453 18609 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:17:23.356998 18609 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:17:23.357127 18609 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:17:23.357255 18609 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:17:23.357343 18609 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:17:23.391369 18609 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.2:36983
I20260924 16:17:23.391459 18737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.2:36983 every 8 connection(s)
I20260924 16:17:23.392701 18609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266323169505-30632-0/minicluster-data/ts-1/data/info.pb
I20260924 16:17:23.398765 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 18609
I20260924 16:17:23.398985 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266323169505-30632-0/minicluster-data/ts-1/wal/instance
I20260924 16:17:23.399566 30632 external_mini_cluster.cc:1428] Setting key d23f60406bb4fbea3b8233f727150254
I20260924 16:17:23.402514 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266323169505-30632-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266323169505-30632-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266323169505-30632-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.234.3:0
--local_ip_for_outbound_sockets=127.29.234.3
--webserver_interface=127.29.234.3
--webserver_port=0
--tserver_master_addrs=127.29.234.62:46225
--builtin_ntp_servers=127.29.234.20:45599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 16:17:23.406626 18607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:23.395527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.2:57239 (local address 127.29.234.62:46225)
0924 16:17:23.395755 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:23.395763 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:23.397293 (+ 1530us) server_negotiation.cc:407] Connection header received
0924 16:17:23.399330 (+ 2037us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:23.399336 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:23.399427 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:23.399540 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:23.400770 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:23.401763 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:23.403449 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:23.404190 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:23.404337 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:23.405708 (+ 1371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:23.405716 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:23.405724 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:17:23.405748 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:23.405789 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:17:23.405806 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:23.405900 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:23.406127 (+ 227us) server_negotiation.cc:299] Negotiation successful
0924 16:17:23.406241 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 16:17:23.407541 18740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:23.396053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:46225 (local address 127.29.234.2:57239)
0924 16:17:23.397102 (+ 1049us) negotiation.cc:107] Waiting for socket to connect
0924 16:17:23.397154 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 16:17:23.398970 (+ 1816us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:17:23.399801 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:17:23.399810 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:17:23.399900 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:17:23.400610 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:23.400627 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:23.401915 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:23.401919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:23.403299 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:23.403311 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:23.404401 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:23.404406 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:23.404610 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:23.405336 (+ 726us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:17:23.405339 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:17:23.405440 (+ 101us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:17:23.405528 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:17:23.405902 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:17:23.405910 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:17:23.405916 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 16:17:23.406084 (+ 168us) client_negotiation.cc:240] Negotiation successful
0924 16:17:23.406402 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":296,"threads_started":1}
I20260924 16:17:23.409053 18738 heartbeater.cc:347] Connected to a master server at 127.29.234.62:46225
I20260924 16:17:23.409415 18738 heartbeater.cc:464] Registering TS with master...
I20260924 16:17:23.410177 18738 heartbeater.cc:511] Master 127.29.234.62:46225 requested a full tablet report, sending...
I20260924 16:17:23.411590 18416 ts_manager.cc:194] Registered new tserver with Master: 6c037edff4604753ab024af3eb260a99 (127.29.234.2:36983)
I20260924 16:17:23.412405 18416 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.234.2:57239
W20260924 16:17:23.739503 18741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:17:23.739843 18741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:17:23.740001 18741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:17:23.748070 18741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:17:23.748169 18741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:17:23.748345 18741 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.234.3
I20260924 16:17:23.757656 18741 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:45599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266323169505-30632-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266323169505-30632-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.234.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.234.3
--webserver_port=0
--tserver_master_addrs=127.29.234.62:46225
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.234.3
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266323169505-30632-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:17:23.758978 18741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:17:23.760771 18741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:17:23.774392 18746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:23.774432 18747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:23.774636 18749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:17:23.775238 18741 server_base.cc:1034] running on GCE node
I20260924 16:17:23.775946 18741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:17:23.776947 18741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:17:23.778199 18741 hybrid_clock.cc:648] HybridClock initialized: now 1790266643778160 us; error 45 us; skew 500 ppm
I20260924 16:17:23.778563 18741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 16:17:23.780871 18741 webserver.cc:460] Webserver started at http://127.29.234.3:38641/ using document root <none> and password file <none>
I20260924 16:17:23.781700 18741 fs_manager.cc:359] Metadata directory not provided
I20260924 16:17:23.781832 18741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:17:23.782151 18741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:17:23.785009 18741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266323169505-30632-0/minicluster-data/ts-2/data/instance:
uuid: "bfe4201b836d438391a5bb7f35c6d9e5"
format_stamp: "Formatted at 2026-09-24 16:17:23 on dist-test-slave-v5j9"
server_key: "9d257624361ccbd8ce5c044309b1b7e5"
server_key_iv: "e2894f6bb3e5eff9c0fcefac24232f2f"
server_key_version: "encryptionkey@0"
I20260924 16:17:23.785851 18741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266323169505-30632-0/minicluster-data/ts-2/wal/instance:
uuid: "bfe4201b836d438391a5bb7f35c6d9e5"
format_stamp: "Formatted at 2026-09-24 16:17:23 on dist-test-slave-v5j9"
server_key: "9d257624361ccbd8ce5c044309b1b7e5"
server_key_iv: "e2894f6bb3e5eff9c0fcefac24232f2f"
server_key_version: "encryptionkey@0"
I20260924 16:17:23.790859 18741 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 16:17:23.794338 18755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:17:23.795740 18741 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 16:17:23.795949 18741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266323169505-30632-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266323169505-30632-0/minicluster-data/ts-2/wal
uuid: "bfe4201b836d438391a5bb7f35c6d9e5"
format_stamp: "Formatted at 2026-09-24 16:17:23 on dist-test-slave-v5j9"
server_key: "9d257624361ccbd8ce5c044309b1b7e5"
server_key_iv: "e2894f6bb3e5eff9c0fcefac24232f2f"
server_key_version: "encryptionkey@0"
I20260924 16:17:23.796200 18741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266323169505-30632-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266323169505-30632-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266323169505-30632-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:17:23.824997 18741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:17:23.826026 18741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:17:23.826388 18741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:17:23.827385 18741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 16:17:23.828917 18741 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 16:17:23.829041 18741 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 16:17:23.829183 18741 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 16:17:23.829278 18741 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 16:17:23.873634 18741 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.3:36809
I20260924 16:17:23.873683 18869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.3:36809 every 8 connection(s)
I20260924 16:17:23.875077 18741 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266323169505-30632-0/minicluster-data/ts-2/data/info.pb
I20260924 16:17:23.878324 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 18741
I20260924 16:17:23.878543 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266323169505-30632-0/minicluster-data/ts-2/wal/instance
I20260924 16:17:23.879011 30632 external_mini_cluster.cc:1428] Setting key b70f5c0e1c36e1f2e4762e69239b9dcf
I20260924 16:17:23.887657 18607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:23.877640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.3:42883 (local address 127.29.234.62:46225)
0924 16:17:23.877891 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:23.877898 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:23.879593 (+ 1695us) server_negotiation.cc:407] Connection header received
0924 16:17:23.881766 (+ 2173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:23.881771 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:23.881858 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:23.881973 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:23.883016 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:23.883744 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:23.885200 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:23.885648 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:23.885743 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:23.886980 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:23.886985 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:23.886988 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:17:23.887005 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:23.887038 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:17:23.887050 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:23.887162 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:23.887338 (+ 176us) server_negotiation.cc:299] Negotiation successful
0924 16:17:23.887416 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260924 16:17:23.888650 18872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:23.878274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:46225 (local address 127.29.234.3:42883)
0924 16:17:23.879400 (+ 1126us) negotiation.cc:107] Waiting for socket to connect
0924 16:17:23.879453 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 16:17:23.881482 (+ 2029us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:17:23.882212 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:17:23.882221 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:17:23.882310 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:17:23.882875 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:23.882891 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:23.883869 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:23.883873 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:23.885076 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:23.885087 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:23.885836 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:23.885841 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:23.886039 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:23.886683 (+ 644us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 16:17:23.886686 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:17:23.886728 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 16:17:23.886810 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:17:23.887165 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:17:23.887173 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:17:23.887178 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 16:17:23.887315 (+ 137us) client_negotiation.cc:240] Negotiation successful
0924 16:17:23.887594 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":380,"threads_started":1}
I20260924 16:17:23.890051 18870 heartbeater.cc:347] Connected to a master server at 127.29.234.62:46225
I20260924 16:17:23.890352 18870 heartbeater.cc:464] Registering TS with master...
I20260924 16:17:23.891080 18870 heartbeater.cc:511] Master 127.29.234.62:46225 requested a full tablet report, sending...
I20260924 16:17:23.892475 18416 ts_manager.cc:194] Registered new tserver with Master: bfe4201b836d438391a5bb7f35c6d9e5 (127.29.234.3:36809)
I20260924 16:17:23.893463 18416 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.234.3:42883
I20260924 16:17:23.894405 30632 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 16:17:23.904114 18607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:23.898443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59036 (local address 127.29.234.62:46225)
0924 16:17:23.898632 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:23.898636 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:23.899105 (+ 469us) server_negotiation.cc:407] Connection header received
0924 16:17:23.899236 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:23.899239 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:23.899306 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:23.899388 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:23.899909 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:23.900706 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:23.902105 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:23.902493 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:23.902585 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:23.903561 (+ 976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:23.903566 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:23.903569 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:17:23.903586 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:23.903631 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:17:23.903643 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:23.903716 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:23.903833 (+ 117us) server_negotiation.cc:299] Negotiation successful
0924 16:17:23.903903 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 16:17:23.907773 18415 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59036:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 16:17:23.915081 18415 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 16:17:23.942384 18883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:23.927897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36870 (local address 127.29.234.2:36983)
0924 16:17:23.930468 (+ 2571us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:23.930480 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:23.930514 (+ 34us) server_negotiation.cc:407] Connection header received
0924 16:17:23.931698 (+ 1184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:23.931707 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:23.931964 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:23.932161 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:17:23.933161 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:23.935687 (+ 2526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:23.939630 (+ 3943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:23.940989 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:23.941125 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:23.941490 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:23.941848 (+ 358us) server_negotiation.cc:299] Negotiation successful
0924 16:17:23.942040 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2361,"thread_start_us":2223,"threads_started":1}
I20260924 16:17:23.942366 18884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:23.927761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.2:36983 (local address 127.0.0.1:36870)
0924 16:17:23.929670 (+ 1909us) negotiation.cc:107] Waiting for socket to connect
0924 16:17:23.929705 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 16:17:23.929869 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:17:23.932454 (+ 2585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:17:23.932461 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:17:23.932509 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:17:23.932964 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:23.932978 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:23.935845 (+ 2867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:23.935851 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:23.939357 (+ 3506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:23.939373 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:23.941179 (+ 1806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:23.941185 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:23.941425 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:23.941578 (+ 153us) client_negotiation.cc:769] Sending connection context
0924 16:17:23.941762 (+ 184us) client_negotiation.cc:240] Negotiation successful
0924 16:17:23.941955 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1658,"mutex_wait_us":72,"thread_start_us":8751,"threads_started":1}
I20260924 16:17:23.946308 18886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:23.930059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.3:36809 (local address 127.0.0.1:43052)
0924 16:17:23.937676 (+ 7617us) negotiation.cc:107] Waiting for socket to connect
0924 16:17:23.937698 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 16:17:23.937855 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:17:23.939038 (+ 1183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:17:23.939046 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:17:23.939080 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:17:23.939600 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:23.939619 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:23.941459 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:23.941464 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:23.943865 (+ 2401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:23.943878 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:23.945536 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:23.945542 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:23.945692 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:23.945728 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 16:17:23.945803 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 16:17:23.945933 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7456,"mutex_wait_us":41,"thread_start_us":7318,"threads_started":1}
I20260924 16:17:23.946456 18887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:23.930729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43052 (local address 127.29.234.3:36809)
0924 16:17:23.937272 (+ 6543us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:23.937280 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:23.938014 (+ 734us) server_negotiation.cc:407] Connection header received
0924 16:17:23.938151 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:23.938163 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:23.938462 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:23.938680 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:17:23.939804 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:23.941248 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:23.944100 (+ 2852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:23.945337 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:23.945483 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:23.945777 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:23.945876 (+ 99us) server_negotiation.cc:299] Negotiation successful
0924 16:17:23.946050 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6349,"thread_start_us":6388,"threads_started":1}
I20260924 16:17:23.946971 18672 tablet_service.cc:1467] Processing CreateTablet for tablet 664ebf736aee48bc9f2ebb65be86ea86 (DEFAULT_TABLE table=test-table [id=9a917512a9ba4c60848f73b22603a3c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:17:23.948004 18882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:23.928040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:41533 (local address 127.0.0.1:48456)
0924 16:17:23.936898 (+ 8858us) negotiation.cc:107] Waiting for socket to connect
0924 16:17:23.936920 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 16:17:23.937047 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:17:23.940421 (+ 3374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:17:23.940428 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:17:23.940457 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:17:23.940778 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:23.940790 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:23.942892 (+ 2102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:23.942898 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:23.945115 (+ 2217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:23.945130 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:23.947279 (+ 2149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:23.947285 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:23.947445 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:23.947483 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 16:17:23.947577 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 16:17:23.947714 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8637,"thread_start_us":1216,"threads_started":1}
I20260924 16:17:23.948150 18885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:23.928630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48456 (local address 127.29.234.1:41533)
0924 16:17:23.939767 (+ 11137us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:23.939776 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:23.939811 (+ 35us) server_negotiation.cc:407] Connection header received
0924 16:17:23.939912 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:23.939920 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:23.940140 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:23.940334 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:17:23.940961 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:23.942714 (+ 1753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:23.945298 (+ 2584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:23.947082 (+ 1784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:23.947222 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:23.947497 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:23.947619 (+ 122us) server_negotiation.cc:299] Negotiation successful
0924 16:17:23.947797 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10971,"thread_start_us":10870,"threads_started":1}
I20260924 16:17:23.948858 18672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 664ebf736aee48bc9f2ebb65be86ea86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:17:23.950181 18804 tablet_service.cc:1467] Processing CreateTablet for tablet 664ebf736aee48bc9f2ebb65be86ea86 (DEFAULT_TABLE table=test-table [id=9a917512a9ba4c60848f73b22603a3c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:17:23.951452 18804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 664ebf736aee48bc9f2ebb65be86ea86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:17:23.958884 18605 heartbeater.cc:503] Master 127.29.234.62:46225 was elected leader, sending a full tablet report...
I20260924 16:17:23.959607 18540 tablet_service.cc:1467] Processing CreateTablet for tablet 664ebf736aee48bc9f2ebb65be86ea86 (DEFAULT_TABLE table=test-table [id=9a917512a9ba4c60848f73b22603a3c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 16:17:23.961127 18540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 664ebf736aee48bc9f2ebb65be86ea86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:17:23.961509 18888 tablet_bootstrap.cc:492] T 664ebf736aee48bc9f2ebb65be86ea86 P 6c037edff4604753ab024af3eb260a99: Bootstrap starting.
I20260924 16:17:23.964556 18888 tablet_bootstrap.cc:654] T 664ebf736aee48bc9f2ebb65be86ea86 P 6c037edff4604753ab024af3eb260a99: Neither blocks nor log segments found. Creating new log.
I20260924 16:17:23.965494 18889 tablet_bootstrap.cc:492] T 664ebf736aee48bc9f2ebb65be86ea86 P bfe4201b836d438391a5bb7f35c6d9e5: Bootstrap starting.
I20260924 16:17:23.966948 18888 log.cc:824] T 664ebf736aee48bc9f2ebb65be86ea86 P 6c037edff4604753ab024af3eb260a99: Log is configured to *not* fsync() on all Append() calls
I20260924 16:17:23.969079 18889 tablet_bootstrap.cc:654] T 664ebf736aee48bc9f2ebb65be86ea86 P bfe4201b836d438391a5bb7f35c6d9e5: Neither blocks nor log segments found. Creating new log.
I20260924 16:17:23.970142 18889 log.cc:824] T 664ebf736aee48bc9f2ebb65be86ea86 P bfe4201b836d438391a5bb7f35c6d9e5: Log is configured to *not* fsync() on all Append() calls
I20260924 16:17:23.971038 18888 tablet_bootstrap.cc:492] T 664ebf736aee48bc9f2ebb65be86ea86 P 6c037edff4604753ab024af3eb260a99: No bootstrap required, opened a new log
I20260924 16:17:23.971405 18888 ts_tablet_manager.cc:1397] T 664ebf736aee48bc9f2ebb65be86ea86 P 6c037edff4604753ab024af3eb260a99: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260924 16:17:23.973098 18889 tablet_bootstrap.cc:492] T 664ebf736aee48bc9f2ebb65be86ea86 P bfe4201b836d438391a5bb7f35c6d9e5: No bootstrap required, opened a new log
I20260924 16:17:23.973420 18889 ts_tablet_manager.cc:1397] T 664ebf736aee48bc9f2ebb65be86ea86 P bfe4201b836d438391a5bb7f35c6d9e5: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260924 16:17:23.975989 18892 tablet_bootstrap.cc:492] T 664ebf736aee48bc9f2ebb65be86ea86 P ff468e18221242c9ab8974e5eef0c229: Bootstrap starting.
I20260924 16:17:23.980408 18892 tablet_bootstrap.cc:654] T 664ebf736aee48bc9f2ebb65be86ea86 P ff468e18221242c9ab8974e5eef0c229: Neither blocks nor log segments found. Creating new log.
I20260924 16:17:23.981093 18889 raft_consensus.cc:348] T 664ebf736aee48bc9f2ebb65be86ea86 P bfe4201b836d438391a5bb7f35c6d9e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c037edff4604753ab024af3eb260a99" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36983 } } peers { permanent_uuid: "ff468e18221242c9ab8974e5eef0c229" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 41533 } } peers { permanent_uuid: "bfe4201b836d438391a5bb7f35c6d9e5" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 36809 } }
I20260924 16:17:23.981537 18889 raft_consensus.cc:374] T 664ebf736aee48bc9f2ebb65be86ea86 P bfe4201b836d438391a5bb7f35c6d9e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:17:23.981703 18889 raft_consensus.cc:729] T 664ebf736aee48bc9f2ebb65be86ea86 P bfe4201b836d438391a5bb7f35c6d9e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfe4201b836d438391a5bb7f35c6d9e5, State: Initialized, Role: FOLLOWER
I20260924 16:17:23.981747 18892 log.cc:824] T 664ebf736aee48bc9f2ebb65be86ea86 P ff468e18221242c9ab8974e5eef0c229: Log is configured to *not* fsync() on all Append() calls
I20260924 16:17:23.982365 18889 consensus_queue.cc:260] T 664ebf736aee48bc9f2ebb65be86ea86 P bfe4201b836d438391a5bb7f35c6d9e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c037edff4604753ab024af3eb260a99" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36983 } } peers { permanent_uuid: "ff468e18221242c9ab8974e5eef0c229" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 41533 } } peers { permanent_uuid: "bfe4201b836d438391a5bb7f35c6d9e5" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 36809 } }
I20260924 16:17:23.982337 18888 raft_consensus.cc:348] T 664ebf736aee48bc9f2ebb65be86ea86 P 6c037edff4604753ab024af3eb260a99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c037edff4604753ab024af3eb260a99" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36983 } } peers { permanent_uuid: "ff468e18221242c9ab8974e5eef0c229" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 41533 } } peers { permanent_uuid: "bfe4201b836d438391a5bb7f35c6d9e5" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 36809 } }
I20260924 16:17:23.982859 18888 raft_consensus.cc:374] T 664ebf736aee48bc9f2ebb65be86ea86 P 6c037edff4604753ab024af3eb260a99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:17:23.983022 18888 raft_consensus.cc:729] T 664ebf736aee48bc9f2ebb65be86ea86 P 6c037edff4604753ab024af3eb260a99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c037edff4604753ab024af3eb260a99, State: Initialized, Role: FOLLOWER
I20260924 16:17:23.983669 18888 consensus_queue.cc:260] T 664ebf736aee48bc9f2ebb65be86ea86 P 6c037edff4604753ab024af3eb260a99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c037edff4604753ab024af3eb260a99" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36983 } } peers { permanent_uuid: "ff468e18221242c9ab8974e5eef0c229" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 41533 } } peers { permanent_uuid: "bfe4201b836d438391a5bb7f35c6d9e5" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 36809 } }
I20260924 16:17:23.984067 18889 ts_tablet_manager.cc:1428] T 664ebf736aee48bc9f2ebb65be86ea86 P bfe4201b836d438391a5bb7f35c6d9e5: Time spent starting tablet: real 0.011s user 0.008s sys 0.002s
I20260924 16:17:23.984972 18738 heartbeater.cc:503] Master 127.29.234.62:46225 was elected leader, sending a full tablet report...
I20260924 16:17:23.985293 18892 tablet_bootstrap.cc:492] T 664ebf736aee48bc9f2ebb65be86ea86 P ff468e18221242c9ab8974e5eef0c229: No bootstrap required, opened a new log
I20260924 16:17:23.985616 18888 ts_tablet_manager.cc:1428] T 664ebf736aee48bc9f2ebb65be86ea86 P 6c037edff4604753ab024af3eb260a99: Time spent starting tablet: real 0.014s user 0.009s sys 0.005s
I20260924 16:17:23.985617 18892 ts_tablet_manager.cc:1397] T 664ebf736aee48bc9f2ebb65be86ea86 P ff468e18221242c9ab8974e5eef0c229: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.002s
I20260924 16:17:23.986063 18870 heartbeater.cc:503] Master 127.29.234.62:46225 was elected leader, sending a full tablet report...
I20260924 16:17:23.994544 18892 raft_consensus.cc:348] T 664ebf736aee48bc9f2ebb65be86ea86 P ff468e18221242c9ab8974e5eef0c229 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c037edff4604753ab024af3eb260a99" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36983 } } peers { permanent_uuid: "ff468e18221242c9ab8974e5eef0c229" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 41533 } } peers { permanent_uuid: "bfe4201b836d438391a5bb7f35c6d9e5" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 36809 } }
I20260924 16:17:23.994977 18892 raft_consensus.cc:374] T 664ebf736aee48bc9f2ebb65be86ea86 P ff468e18221242c9ab8974e5eef0c229 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:17:23.995102 18892 raft_consensus.cc:729] T 664ebf736aee48bc9f2ebb65be86ea86 P ff468e18221242c9ab8974e5eef0c229 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff468e18221242c9ab8974e5eef0c229, State: Initialized, Role: FOLLOWER
I20260924 16:17:23.995669 18892 consensus_queue.cc:260] T 664ebf736aee48bc9f2ebb65be86ea86 P ff468e18221242c9ab8974e5eef0c229 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c037edff4604753ab024af3eb260a99" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36983 } } peers { permanent_uuid: "ff468e18221242c9ab8974e5eef0c229" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 41533 } } peers { permanent_uuid: "bfe4201b836d438391a5bb7f35c6d9e5" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 36809 } }
I20260924 16:17:23.997131 18892 ts_tablet_manager.cc:1428] T 664ebf736aee48bc9f2ebb65be86ea86 P ff468e18221242c9ab8974e5eef0c229: Time spent starting tablet: real 0.011s user 0.009s sys 0.002s
I20260924 16:17:24.040056 18894 raft_consensus.cc:482] T 664ebf736aee48bc9f2ebb65be86ea86 P bfe4201b836d438391a5bb7f35c6d9e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 16:17:24.040345 18894 raft_consensus.cc:504] T 664ebf736aee48bc9f2ebb65be86ea86 P bfe4201b836d438391a5bb7f35c6d9e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c037edff4604753ab024af3eb260a99" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36983 } } peers { permanent_uuid: "ff468e18221242c9ab8974e5eef0c229" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 41533 } } peers { permanent_uuid: "bfe4201b836d438391a5bb7f35c6d9e5" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 36809 } }
I20260924 16:17:24.042390 18894 leader_election.cc:288] T 664ebf736aee48bc9f2ebb65be86ea86 P bfe4201b836d438391a5bb7f35c6d9e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c037edff4604753ab024af3eb260a99 (127.29.234.2:36983), ff468e18221242c9ab8974e5eef0c229 (127.29.234.1:41533)
I20260924 16:17:24.049033 18872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:24.042681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:41533 (local address 127.29.234.3:45665)
0924 16:17:24.042893 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0924 16:17:24.042914 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 16:17:24.043055 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:17:24.043509 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:17:24.043519 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:17:24.043546 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:17:24.043750 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:24.043765 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:24.044986 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:24.044989 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:24.047108 (+ 2119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:24.047123 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:24.048481 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:24.048485 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:24.048600 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:24.048624 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 16:17:24.048696 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 16:17:24.048789 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260924 16:17:24.049139 18885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:24.042845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.3:45665 (local address 127.29.234.1:41533)
0924 16:17:24.043064 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:24.043073 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:24.043105 (+ 32us) server_negotiation.cc:407] Connection header received
0924 16:17:24.043204 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:24.043212 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:24.043320 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:24.043424 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:17:24.043934 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:24.044858 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:24.047267 (+ 2409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:24.048331 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:24.048493 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:24.048565 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:24.048746 (+ 181us) server_negotiation.cc:299] Negotiation successful
0924 16:17:24.048884 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 16:17:24.049939 18897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:24.042781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.2:36983 (local address 127.29.234.3:50259)
0924 16:17:24.043685 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0924 16:17:24.043704 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 16:17:24.043836 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:17:24.044170 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:17:24.044172 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:17:24.044191 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 16:17:24.044475 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:24.044483 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:24.045655 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:24.045661 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:24.047564 (+ 1903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:17:24.047574 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:17:24.049116 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:17:24.049124 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:17:24.049314 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:24.049364 (+ 50us) client_negotiation.cc:769] Sending connection context
0924 16:17:24.049508 (+ 144us) client_negotiation.cc:240] Negotiation successful
0924 16:17:24.049656 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":773,"thread_start_us":511,"threads_started":1}
I20260924 16:17:24.050422 18560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "664ebf736aee48bc9f2ebb65be86ea86" candidate_uuid: "bfe4201b836d438391a5bb7f35c6d9e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff468e18221242c9ab8974e5eef0c229" is_pre_election: true
I20260924 16:17:24.051045 18560 raft_consensus.cc:2454] T 664ebf736aee48bc9f2ebb65be86ea86 P ff468e18221242c9ab8974e5eef0c229 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfe4201b836d438391a5bb7f35c6d9e5 in term 0.
I20260924 16:17:24.051349 18883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:24.043047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.234.3:50259 (local address 127.29.234.2:36983)
0924 16:17:24.043284 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:24.043291 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:24.043802 (+ 511us) server_negotiation.cc:407] Connection header received
0924 16:17:24.043965 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:24.043971 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:24.044059 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:24.044151 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 16:17:24.044631 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:24.045491 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:24.047730 (+ 2239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:24.048923 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:24.049064 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:24.049125 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:24.049572 (+ 447us) server_negotiation.cc:299] Negotiation successful
0924 16:17:24.049711 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 16:17:24.051676 18757 leader_election.cc:302] T 664ebf736aee48bc9f2ebb65be86ea86 P bfe4201b836d438391a5bb7f35c6d9e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bfe4201b836d438391a5bb7f35c6d9e5, ff468e18221242c9ab8974e5eef0c229; no voters:
I20260924 16:17:24.052074 18894 raft_consensus.cc:2781] T 664ebf736aee48bc9f2ebb65be86ea86 P bfe4201b836d438391a5bb7f35c6d9e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 16:17:24.052242 18894 raft_consensus.cc:482] T 664ebf736aee48bc9f2ebb65be86ea86 P bfe4201b836d438391a5bb7f35c6d9e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 16:17:24.052348 18894 raft_consensus.cc:3037] T 664ebf736aee48bc9f2ebb65be86ea86 P bfe4201b836d438391a5bb7f35c6d9e5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:17:24.052279 18693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "664ebf736aee48bc9f2ebb65be86ea86" candidate_uuid: "bfe4201b836d438391a5bb7f35c6d9e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c037edff4604753ab024af3eb260a99" is_pre_election: true
I20260924 16:17:24.052830 18693 raft_consensus.cc:2454] T 664ebf736aee48bc9f2ebb65be86ea86 P 6c037edff4604753ab024af3eb260a99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfe4201b836d438391a5bb7f35c6d9e5 in term 0.
I20260924 16:17:24.054908 18894 raft_consensus.cc:504] T 664ebf736aee48bc9f2ebb65be86ea86 P bfe4201b836d438391a5bb7f35c6d9e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c037edff4604753ab024af3eb260a99" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36983 } } peers { permanent_uuid: "ff468e18221242c9ab8974e5eef0c229" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 41533 } } peers { permanent_uuid: "bfe4201b836d438391a5bb7f35c6d9e5" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 36809 } }
I20260924 16:17:24.055516 18894 leader_election.cc:288] T 664ebf736aee48bc9f2ebb65be86ea86 P bfe4201b836d438391a5bb7f35c6d9e5 [CANDIDATE]: Term 1 election: Requested vote from peers 6c037edff4604753ab024af3eb260a99 (127.29.234.2:36983), ff468e18221242c9ab8974e5eef0c229 (127.29.234.1:41533)
I20260924 16:17:24.055979 18693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "664ebf736aee48bc9f2ebb65be86ea86" candidate_uuid: "bfe4201b836d438391a5bb7f35c6d9e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c037edff4604753ab024af3eb260a99"
I20260924 16:17:24.056182 18693 raft_consensus.cc:3037] T 664ebf736aee48bc9f2ebb65be86ea86 P 6c037edff4604753ab024af3eb260a99 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:17:24.056066 18560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "664ebf736aee48bc9f2ebb65be86ea86" candidate_uuid: "bfe4201b836d438391a5bb7f35c6d9e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff468e18221242c9ab8974e5eef0c229"
I20260924 16:17:24.056383 18560 raft_consensus.cc:3037] T 664ebf736aee48bc9f2ebb65be86ea86 P ff468e18221242c9ab8974e5eef0c229 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:17:24.058526 18693 raft_consensus.cc:2454] T 664ebf736aee48bc9f2ebb65be86ea86 P 6c037edff4604753ab024af3eb260a99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfe4201b836d438391a5bb7f35c6d9e5 in term 1.
I20260924 16:17:24.058749 18560 raft_consensus.cc:2454] T 664ebf736aee48bc9f2ebb65be86ea86 P ff468e18221242c9ab8974e5eef0c229 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfe4201b836d438391a5bb7f35c6d9e5 in term 1.
I20260924 16:17:24.059000 18759 leader_election.cc:302] T 664ebf736aee48bc9f2ebb65be86ea86 P bfe4201b836d438391a5bb7f35c6d9e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c037edff4604753ab024af3eb260a99, bfe4201b836d438391a5bb7f35c6d9e5; no voters:
I20260924 16:17:24.059325 18894 raft_consensus.cc:2781] T 664ebf736aee48bc9f2ebb65be86ea86 P bfe4201b836d438391a5bb7f35c6d9e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:17:24.059767 18894 raft_consensus.cc:686] T 664ebf736aee48bc9f2ebb65be86ea86 P bfe4201b836d438391a5bb7f35c6d9e5 [term 1 LEADER]: Becoming Leader. State: Replica: bfe4201b836d438391a5bb7f35c6d9e5, State: Running, Role: LEADER
I20260924 16:17:24.060242 18894 consensus_queue.cc:237] T 664ebf736aee48bc9f2ebb65be86ea86 P bfe4201b836d438391a5bb7f35c6d9e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c037edff4604753ab024af3eb260a99" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36983 } } peers { permanent_uuid: "ff468e18221242c9ab8974e5eef0c229" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 41533 } } peers { permanent_uuid: "bfe4201b836d438391a5bb7f35c6d9e5" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 36809 } }
I20260924 16:17:24.064857 18416 catalog_manager.cc:5495] T 664ebf736aee48bc9f2ebb65be86ea86 P bfe4201b836d438391a5bb7f35c6d9e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to bfe4201b836d438391a5bb7f35c6d9e5 (127.29.234.3). New cstate: current_term: 1 leader_uuid: "bfe4201b836d438391a5bb7f35c6d9e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c037edff4604753ab024af3eb260a99" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff468e18221242c9ab8974e5eef0c229" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 41533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bfe4201b836d438391a5bb7f35c6d9e5" member_type: VOTER last_known_addr { host: "127.29.234.3" port: 36809 } health_report { overall_health: HEALTHY } } }
I20260924 16:17:24.092202 18887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:24.087189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43064 (local address 127.29.234.3:36809)
0924 16:17:24.087369 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:24.087374 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:24.087392 (+ 18us) server_negotiation.cc:407] Connection header received
0924 16:17:24.087482 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:24.087485 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:24.087560 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:24.087703 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:17:24.088025 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:24.088729 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:24.090147 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:24.090556 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:24.090658 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:24.091093 (+ 435us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:17:24.091633 (+ 540us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:17:24.091756 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:24.091858 (+ 102us) server_negotiation.cc:299] Negotiation successful
0924 16:17:24.091922 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 16:17:24.100677 18560 raft_consensus.cc:1261] T 664ebf736aee48bc9f2ebb65be86ea86 P ff468e18221242c9ab8974e5eef0c229 [term 1 FOLLOWER]: Refusing update from remote peer bfe4201b836d438391a5bb7f35c6d9e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:17:24.100764 18693 raft_consensus.cc:1261] T 664ebf736aee48bc9f2ebb65be86ea86 P 6c037edff4604753ab024af3eb260a99 [term 1 FOLLOWER]: Refusing update from remote peer bfe4201b836d438391a5bb7f35c6d9e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:17:24.101773 18898 consensus_queue.cc:1035] T 664ebf736aee48bc9f2ebb65be86ea86 P bfe4201b836d438391a5bb7f35c6d9e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff468e18221242c9ab8974e5eef0c229" member_type: VOTER last_known_addr { host: "127.29.234.1" port: 41533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:17:24.102121 18894 consensus_queue.cc:1035] T 664ebf736aee48bc9f2ebb65be86ea86 P bfe4201b836d438391a5bb7f35c6d9e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c037edff4604753ab024af3eb260a99" member_type: VOTER last_known_addr { host: "127.29.234.2" port: 36983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:17:24.127010 18415 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:17:24.127292 18415 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 16:17:24.130885 18415 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1edc27698e484944b1f945e20140925a: Sending DeleteTablet for 3 replicas of tablet 664ebf736aee48bc9f2ebb65be86ea86
I20260924 16:17:24.132014 18672 tablet_service.cc:1514] Processing DeleteTablet for tablet 664ebf736aee48bc9f2ebb65be86ea86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:17:24 UTC) from {username='slave'} at 127.0.0.1:36870
I20260924 16:17:24.132102 18540 tablet_service.cc:1514] Processing DeleteTablet for tablet 664ebf736aee48bc9f2ebb65be86ea86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:17:24 UTC) from {username='slave'} at 127.0.0.1:48456
I20260924 16:17:24.132277 18804 tablet_service.cc:1514] Processing DeleteTablet for tablet 664ebf736aee48bc9f2ebb65be86ea86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 16:17:24 UTC) from {username='slave'} at 127.0.0.1:43052
I20260924 16:17:24.134315 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 18476
I20260924 16:17:24.144855 18913 tablet_replica.cc:330] T 664ebf736aee48bc9f2ebb65be86ea86 P bfe4201b836d438391a5bb7f35c6d9e5: stopping tablet replica
I20260924 16:17:24.145303 18913 raft_consensus.cc:2229] T 664ebf736aee48bc9f2ebb65be86ea86 P bfe4201b836d438391a5bb7f35c6d9e5 [term 1 LEADER]: Raft consensus shutting down.
I20260924 16:17:24.147318 18913 raft_consensus.cc:2258] T 664ebf736aee48bc9f2ebb65be86ea86 P bfe4201b836d438391a5bb7f35c6d9e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:17:24.150489 18913 ts_tablet_manager.cc:1905] T 664ebf736aee48bc9f2ebb65be86ea86 P bfe4201b836d438391a5bb7f35c6d9e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:17:24.152904 18911 tablet_replica.cc:330] T 664ebf736aee48bc9f2ebb65be86ea86 P 6c037edff4604753ab024af3eb260a99: stopping tablet replica
I20260924 16:17:24.153930 18912 tablet_replica.cc:330] T 664ebf736aee48bc9f2ebb65be86ea86 P ff468e18221242c9ab8974e5eef0c229: stopping tablet replica
I20260924 16:17:24.154385 18912 raft_consensus.cc:2229] T 664ebf736aee48bc9f2ebb65be86ea86 P ff468e18221242c9ab8974e5eef0c229 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:17:24.154922 18912 raft_consensus.cc:2258] T 664ebf736aee48bc9f2ebb65be86ea86 P ff468e18221242c9ab8974e5eef0c229 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:17:24.155375 18911 raft_consensus.cc:2229] T 664ebf736aee48bc9f2ebb65be86ea86 P 6c037edff4604753ab024af3eb260a99 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 16:17:24.155845 18911 raft_consensus.cc:2258] T 664ebf736aee48bc9f2ebb65be86ea86 P 6c037edff4604753ab024af3eb260a99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 16:17:24.155328 18885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:24.136071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48458 (local address 127.29.234.1:41533)
0924 16:17:24.136296 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:24.136305 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:24.141775 (+ 5470us) server_negotiation.cc:407] Connection header received
0924 16:17:24.142625 (+ 850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:24.142635 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:24.142799 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:24.142915 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:24.146805 (+ 3890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:24.148216 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:24.150593 (+ 2377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:24.151508 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:24.151635 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:24.153444 (+ 1809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:24.153458 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:24.153483 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:17:24.154312 (+ 829us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:24.154388 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:17:24.154415 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:24.154520 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:24.154777 (+ 257us) server_negotiation.cc:299] Negotiation successful
0924 16:17:24.154907 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 16:17:24.158095 18600 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:17:24.158192 18911 ts_tablet_manager.cc:1905] T 664ebf736aee48bc9f2ebb65be86ea86 P 6c037edff4604753ab024af3eb260a99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 16:17:24.165171 18913 ts_tablet_manager.cc:1918] T 664ebf736aee48bc9f2ebb65be86ea86 P bfe4201b836d438391a5bb7f35c6d9e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:17:24.165520 18913 log.cc:1196] T 664ebf736aee48bc9f2ebb65be86ea86 P bfe4201b836d438391a5bb7f35c6d9e5: Deleting WAL directory at /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266323169505-30632-0/minicluster-data/ts-2/wal/wals/664ebf736aee48bc9f2ebb65be86ea86
I20260924 16:17:24.165868 18911 ts_tablet_manager.cc:1918] T 664ebf736aee48bc9f2ebb65be86ea86 P 6c037edff4604753ab024af3eb260a99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 16:17:24.166134 18911 log.cc:1196] T 664ebf736aee48bc9f2ebb65be86ea86 P 6c037edff4604753ab024af3eb260a99: Deleting WAL directory at /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790266323169505-30632-0/minicluster-data/ts-1/wal/wals/664ebf736aee48bc9f2ebb65be86ea86
I20260924 16:17:24.166227 18913 ts_tablet_manager.cc:1939] T 664ebf736aee48bc9f2ebb65be86ea86 P bfe4201b836d438391a5bb7f35c6d9e5: Deleting consensus metadata
I20260924 16:17:24.166728 18911 ts_tablet_manager.cc:1939] T 664ebf736aee48bc9f2ebb65be86ea86 P 6c037edff4604753ab024af3eb260a99: Deleting consensus metadata
I20260924 16:17:24.168275 18401 catalog_manager.cc:4841] TS bfe4201b836d438391a5bb7f35c6d9e5 (127.29.234.3:36809): tablet 664ebf736aee48bc9f2ebb65be86ea86 (table test-table [id=9a917512a9ba4c60848f73b22603a3c5]) successfully deleted
I20260924 16:17:24.168527 18403 catalog_manager.cc:4841] TS 6c037edff4604753ab024af3eb260a99 (127.29.234.2:36983): tablet 664ebf736aee48bc9f2ebb65be86ea86 (table test-table [id=9a917512a9ba4c60848f73b22603a3c5]) successfully deleted
I20260924 16:17:24.262241 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 18476
I20260924 16:17:24.280514 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 18609
W20260924 16:17:24.280530 18401 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.234.1:41533 (error 108)
I20260924 16:17:24.282944 18882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:24.282443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:41533 (local address 127.0.0.1:48472)
0924 16:17:24.282642 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0924 16:17:24.282766 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.1:41533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260924 16:17:24.283815 18401 catalog_manager.cc:4584] TS ff468e18221242c9ab8974e5eef0c229 (127.29.234.1:41533): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 664ebf736aee48bc9f2ebb65be86ea86: Network error: Client connection negotiation failed: client connection to 127.29.234.1:41533: connect: Connection refused (error 111)
I20260924 16:17:24.288681 18883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:24.281672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36876 (local address 127.29.234.2:36983)
0924 16:17:24.281876 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:24.281885 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:24.281909 (+ 24us) server_negotiation.cc:407] Connection header received
0924 16:17:24.281978 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:24.281985 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:24.282084 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:24.282178 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:24.282693 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:24.283586 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:24.285502 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:24.285914 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:24.286009 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:24.287259 (+ 1250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:24.287268 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:24.287284 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:17:24.287899 (+ 615us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:24.287950 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:17:24.287969 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:24.288063 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:24.288222 (+ 159us) server_negotiation.cc:299] Negotiation successful
0924 16:17:24.288317 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 16:17:24.289438 18733 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:17:24.337688 18882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:24.337021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:41533 (local address 127.0.0.1:48484)
0924 16:17:24.337307 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0924 16:17:24.337465 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.1:41533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260924 16:17:24.338960 18882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:24.338535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:41533 (local address 127.0.0.1:48490)
0924 16:17:24.338706 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 16:17:24.338798 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.1:41533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260924 16:17:24.379877 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 18609
I20260924 16:17:24.398180 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 18741
I20260924 16:17:24.400213 18882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:24.399603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:41533 (local address 127.0.0.1:48502)
0924 16:17:24.399931 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0924 16:17:24.400027 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.1:41533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187}
I20260924 16:17:24.401558 18882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:24.401122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:41533 (local address 127.0.0.1:48504)
0924 16:17:24.401327 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0924 16:17:24.401423 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.1:41533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260924 16:17:24.406394 18887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:24.398936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43070 (local address 127.29.234.3:36809)
0924 16:17:24.399118 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:24.399125 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:24.399263 (+ 138us) server_negotiation.cc:407] Connection header received
0924 16:17:24.399522 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:24.399528 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:24.399677 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:24.399805 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:24.400526 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:24.401802 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:24.403069 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:24.403527 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:24.403667 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:24.404935 (+ 1268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:24.404944 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:24.404958 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:17:24.405616 (+ 658us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:24.405662 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:17:24.405680 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:24.405818 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:24.405921 (+ 103us) server_negotiation.cc:299] Negotiation successful
0924 16:17:24.406018 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 16:17:24.407014 18865 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:17:24.482416 18882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:24.481899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:41533 (local address 127.0.0.1:48514)
0924 16:17:24.482126 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0924 16:17:24.482253 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.1:41533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260924 16:17:24.483538 18882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:24.483124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.1:41533 (local address 127.0.0.1:48530)
0924 16:17:24.483327 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0924 16:17:24.483433 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.1:41533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260924 16:17:24.529960 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 18741
I20260924 16:17:24.549829 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 18384
I20260924 16:17:24.550942 18447 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:17:24.662168 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 18384
2026-09-24T16:17:24Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2823 ms)
[----------] 6 tests from EncryptionPolicyTest (21488 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790266323169505-30632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:17:24 dist-test-slave-v5j9 krb5kdc[18923](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:17:24 dist-test-slave-v5j9 krb5kdc[18923](info): set up 2 sockets
Sep 24 16:17:24 dist-test-slave-v5j9 krb5kdc[18923](info): commencing operation
krb5kdc: starting...
W20260924 16:17:25.793551 30632 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Sep 24 16:17:25 dist-test-slave-v5j9 krb5kdc[18923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266645, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:17:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:17:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790266323169505-30632-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790266323169505-30632-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 16:17:26.075278 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790266323169505-30632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790266323169505-30632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790266323169505-30632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32943
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790266323169505-30632-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32943
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:17:26.413688 18936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:17:26.414009 18936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:17:26.414193 18936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:17:26.421876 18936 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:17:26.421993 18936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:17:26.422096 18936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:17:26.422199 18936 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:17:26.422271 18936 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:17:26.432277 18936 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790266323169505-30632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790266323169505-30632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32943
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790266323169505-30632-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790266323169505-30632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:17:26.433810 18936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:17:26.435551 18936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:17:26.447757 18941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:26.447728 18942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:26.448103 18944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:17:26.448607 18936 server_base.cc:1034] running on GCE node
I20260924 16:17:26.449550 18936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:17:26.451035 18936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:17:26.452241 18936 hybrid_clock.cc:648] HybridClock initialized: now 1790266646452184 us; error 56 us; skew 500 ppm
I20260924 16:17:26.452626 18936 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:17:26 dist-test-slave-v5j9 krb5kdc[18923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266646, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:17:26.455883 18936 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 16:17:26.457396 18936 webserver.cc:460] Webserver started at http://127.0.0.1:40093/ using document root <none> and password file <none>
I20260924 16:17:26.458102 18936 fs_manager.cc:359] Metadata directory not provided
I20260924 16:17:26.458258 18936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:17:26.458573 18936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:17:26.461668 18936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790266323169505-30632-0/minicluster-data/master-0/data/instance:
uuid: "0ce67f3ac61a49829842ac129e156597"
format_stamp: "Formatted at 2026-09-24 16:17:26 on dist-test-slave-v5j9"
server_key: "37ee9b372635bce06ccb536a7880687c"
server_key_iv: "d9fa5d7a05fb17b9bf2372b4d9ab352e"
server_key_version: "encryptionkey@0"
I20260924 16:17:26.462491 18936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790266323169505-30632-0/minicluster-data/master-0/wal/instance:
uuid: "0ce67f3ac61a49829842ac129e156597"
format_stamp: "Formatted at 2026-09-24 16:17:26 on dist-test-slave-v5j9"
server_key: "37ee9b372635bce06ccb536a7880687c"
server_key_iv: "d9fa5d7a05fb17b9bf2372b4d9ab352e"
server_key_version: "encryptionkey@0"
I20260924 16:17:26.468116 18936 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 16:17:26.471974 18951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:17:26.473464 18936 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 16:17:26.473673 18936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790266323169505-30632-0/minicluster-data/master-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790266323169505-30632-0/minicluster-data/master-0/wal
uuid: "0ce67f3ac61a49829842ac129e156597"
format_stamp: "Formatted at 2026-09-24 16:17:26 on dist-test-slave-v5j9"
server_key: "37ee9b372635bce06ccb536a7880687c"
server_key_iv: "d9fa5d7a05fb17b9bf2372b4d9ab352e"
server_key_version: "encryptionkey@0"
I20260924 16:17:26.473884 18936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790266323169505-30632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790266323169505-30632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790266323169505-30632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:17:26.513481 18936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:17:26.517550 18936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:17:26.517843 18936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:17:26.536899 18936 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32943
I20260924 16:17:26.536897 19002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32943 every 8 connection(s)
I20260924 16:17:26.538566 18936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
I20260924 16:17:26.540628 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 18936
I20260924 16:17:26.540884 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790266323169505-30632-0/minicluster-data/master-0/wal/instance
I20260924 16:17:26.541790 30632 external_mini_cluster.cc:1428] Setting key 1dc4b11d0c1f96ca46e1794052aa4256
I20260924 16:17:26.543732 19003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 16:17:26 dist-test-slave-v5j9 krb5kdc[18923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266645, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 16:17:26.558753 19003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ce67f3ac61a49829842ac129e156597: Bootstrap starting.
I20260924 16:17:26.562611 19003 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ce67f3ac61a49829842ac129e156597: Neither blocks nor log segments found. Creating new log.
I20260924 16:17:26.563807 19003 log.cc:824] T 00000000000000000000000000000000 P 0ce67f3ac61a49829842ac129e156597: Log is configured to *not* fsync() on all Append() calls
I20260924 16:17:26.567257 19003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ce67f3ac61a49829842ac129e156597: No bootstrap required, opened a new log
I20260924 16:17:26.567443 19006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:26.544025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59819 (local address 127.0.0.1:32943)
0924 16:17:26.544892 (+ 867us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:26.544905 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:26.544949 (+ 44us) server_negotiation.cc:407] Connection header received
0924 16:17:26.546786 (+ 1837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:26.546813 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:26.547275 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:26.547639 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:26.549545 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:26.551034 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:26.552857 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:26.553678 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:26.553832 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:26.558866 (+ 5034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:26.558875 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:26.558895 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:17:26.558992 (+ 97us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:26.562544 (+ 3552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:17:26.563524 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:17:26.563532 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:17:26.563539 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:17:26.563734 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:17:26.564334 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:17:26.564344 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:17:26.564346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:17:26.564998 (+ 652us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:26.565397 (+ 399us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:26.565961 (+ 564us) server_negotiation.cc:299] Negotiation successful
0924 16:17:26.566354 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":397,"threads_started":1}
I20260924 16:17:26.572065 30632 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:17:26.576004 19003 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ce67f3ac61a49829842ac129e156597 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ce67f3ac61a49829842ac129e156597" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32943 } }
I20260924 16:17:26.576435 19003 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ce67f3ac61a49829842ac129e156597 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:17:26.576560 19003 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ce67f3ac61a49829842ac129e156597 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ce67f3ac61a49829842ac129e156597, State: Initialized, Role: FOLLOWER
I20260924 16:17:26.577333 19003 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ce67f3ac61a49829842ac129e156597 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ce67f3ac61a49829842ac129e156597" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32943 } }
I20260924 16:17:26.577708 19003 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ce67f3ac61a49829842ac129e156597 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:17:26.577911 19003 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ce67f3ac61a49829842ac129e156597 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:17:26.578135 19003 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ce67f3ac61a49829842ac129e156597 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:17:26.580886 19003 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ce67f3ac61a49829842ac129e156597 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ce67f3ac61a49829842ac129e156597" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32943 } }
I20260924 16:17:26.581537 19003 leader_election.cc:302] T 00000000000000000000000000000000 P 0ce67f3ac61a49829842ac129e156597 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ce67f3ac61a49829842ac129e156597; no voters:
I20260924 16:17:26.582160 19003 leader_election.cc:288] T 00000000000000000000000000000000 P 0ce67f3ac61a49829842ac129e156597 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:17:26.582304 19013 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ce67f3ac61a49829842ac129e156597 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:17:26.584292 19003 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ce67f3ac61a49829842ac129e156597 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:17:26.589244 19013 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ce67f3ac61a49829842ac129e156597 [term 1 LEADER]: Becoming Leader. State: Replica: 0ce67f3ac61a49829842ac129e156597, State: Running, Role: LEADER
I20260924 16:17:26.589852 19013 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ce67f3ac61a49829842ac129e156597 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ce67f3ac61a49829842ac129e156597" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32943 } }
I20260924 16:17:26.592761 19014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ce67f3ac61a49829842ac129e156597 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ce67f3ac61a49829842ac129e156597" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ce67f3ac61a49829842ac129e156597" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32943 } } }
I20260924 16:17:26.593187 19014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ce67f3ac61a49829842ac129e156597 [sys.catalog]: This master's current role is: LEADER
I20260924 16:17:26.593719 19015 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ce67f3ac61a49829842ac129e156597 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ce67f3ac61a49829842ac129e156597. Latest consensus state: current_term: 1 leader_uuid: "0ce67f3ac61a49829842ac129e156597" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ce67f3ac61a49829842ac129e156597" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32943 } } }
I20260924 16:17:26.594079 19015 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ce67f3ac61a49829842ac129e156597 [sys.catalog]: This master's current role is: LEADER
I20260924 16:17:26.599005 19006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:26.577171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48309 (local address 127.0.0.1:32943)
0924 16:17:26.577452 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:26.577461 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:26.577878 (+ 417us) server_negotiation.cc:407] Connection header received
0924 16:17:26.578242 (+ 364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:26.578249 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:26.578349 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:26.578476 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:26.579893 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:26.581006 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:26.582776 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:26.583798 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:26.583923 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:26.588219 (+ 4296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:26.588229 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:26.588236 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:17:26.588299 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:26.592872 (+ 4573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:17:26.594320 (+ 1448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:17:26.594330 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:17:26.594336 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:17:26.594595 (+ 259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:17:26.595401 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:17:26.595410 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:17:26.595416 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:17:26.595936 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:26.596340 (+ 404us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:26.598134 (+ 1794us) server_negotiation.cc:299] Negotiation successful
0924 16:17:26.598426 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 16:17:26.605559 19024 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:17:26.610420 19024 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:17:26.619575 19024 catalog_manager.cc:1347] Generated new cluster ID: d5ba7da874b547118ea81d9444168e8f
I20260924 16:17:26.619705 19024 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:17:26.641546 19024 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:17:26.643442 19024 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:17:26.650765 19024 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ce67f3ac61a49829842ac129e156597: Generated new TSK 0
I20260924 16:17:26.651655 19024 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:17:26.658355 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 18936
I20260924 16:17:26.660276 18998 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:17:26.752969 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 18936
W20260924 16:17:26.771901 18921 connection.cc:535] client connection to 127.0.0.1:32943 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32943): Connection reset by peer (error 104)
2026-09-24T16:17:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2080 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790266323169505-30632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:17:26 dist-test-slave-v5j9 krb5kdc[19036](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:17:26 dist-test-slave-v5j9 krb5kdc[19036](info): set up 2 sockets
Sep 24 16:17:26 dist-test-slave-v5j9 krb5kdc[19036](info): commencing operation
krb5kdc: starting...
W20260924 16:17:27.877542 30632 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Sep 24 16:17:28 dist-test-slave-v5j9 krb5kdc[19036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:17:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:17:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790266323169505-30632-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790266323169505-30632-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 16:17:28.148620 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790266323169505-30632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790266323169505-30632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790266323169505-30632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32849
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790266323169505-30632-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32849
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:17:28.497884 19050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:17:28.498221 19050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:17:28.498318 19050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:17:28.506042 19050 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:17:28.506192 19050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:17:28.506247 19050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:17:28.506304 19050 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:17:28.506350 19050 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:17:28.516695 19050 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790266323169505-30632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790266323169505-30632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32849
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790266323169505-30632-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790266323169505-30632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:17:28.518075 19050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:17:28.519387 19050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:17:28.531622 19058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:28.531589 19055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:28.531562 19056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:17:28.532461 19050 server_base.cc:1034] running on GCE node
I20260924 16:17:28.533427 19050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:17:28.535010 19050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:17:28.536267 19050 hybrid_clock.cc:648] HybridClock initialized: now 1790266648536222 us; error 57 us; skew 500 ppm
I20260924 16:17:28.536680 19050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:17:28 dist-test-slave-v5j9 krb5kdc[19036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266648, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:17:28.551241 19050 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 16:17:28.553184 19050 webserver.cc:460] Webserver started at http://127.0.0.1:46735/ using document root <none> and password file <none>
I20260924 16:17:28.553999 19050 fs_manager.cc:359] Metadata directory not provided
I20260924 16:17:28.554124 19050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:17:28.554411 19050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:17:28.557739 19050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790266323169505-30632-0/minicluster-data/master-0/data/instance:
uuid: "7199cfd144ed4859b9d91df4f435f89e"
format_stamp: "Formatted at 2026-09-24 16:17:28 on dist-test-slave-v5j9"
server_key: "d1e52a989ffefbb5880ca1bf8cacbc8c"
server_key_iv: "d23ac31ff54a6bad76c0ceed9b4f0d5d"
server_key_version: "encryptionkey@0"
I20260924 16:17:28.558531 19050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790266323169505-30632-0/minicluster-data/master-0/wal/instance:
uuid: "7199cfd144ed4859b9d91df4f435f89e"
format_stamp: "Formatted at 2026-09-24 16:17:28 on dist-test-slave-v5j9"
server_key: "d1e52a989ffefbb5880ca1bf8cacbc8c"
server_key_iv: "d23ac31ff54a6bad76c0ceed9b4f0d5d"
server_key_version: "encryptionkey@0"
I20260924 16:17:28.564322 19050 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 16:17:28.568368 19065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:17:28.569893 19050 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 16:17:28.570102 19050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790266323169505-30632-0/minicluster-data/master-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790266323169505-30632-0/minicluster-data/master-0/wal
uuid: "7199cfd144ed4859b9d91df4f435f89e"
format_stamp: "Formatted at 2026-09-24 16:17:28 on dist-test-slave-v5j9"
server_key: "d1e52a989ffefbb5880ca1bf8cacbc8c"
server_key_iv: "d23ac31ff54a6bad76c0ceed9b4f0d5d"
server_key_version: "encryptionkey@0"
I20260924 16:17:28.570339 19050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790266323169505-30632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790266323169505-30632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790266323169505-30632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:17:28.585901 19050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:17:28.589965 19050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:17:28.590266 19050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:17:28.608808 19050 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32849
I20260924 16:17:28.608839 19117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32849 every 8 connection(s)
I20260924 16:17:28.610281 19050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
I20260924 16:17:28.614012 19118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:17:28.616446 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 19050
I20260924 16:17:28.616647 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790266323169505-30632-0/minicluster-data/master-0/wal/instance
I20260924 16:17:28.617178 30632 external_mini_cluster.cc:1428] Setting key fbcf00b2b5d4d19fa2268b95a68696a6
I20260924 16:17:28.626752 19118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7199cfd144ed4859b9d91df4f435f89e: Bootstrap starting.
I20260924 16:17:28.630512 19118 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7199cfd144ed4859b9d91df4f435f89e: Neither blocks nor log segments found. Creating new log.
I20260924 16:17:28.630966 19121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:28.619092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34381 (local address 127.0.0.1:32849)
0924 16:17:28.619951 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:28.619964 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:28.620012 (+ 48us) server_negotiation.cc:407] Connection header received
0924 16:17:28.621782 (+ 1770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:28.621807 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:28.622212 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:28.622527 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:28.623189 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:28.624545 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:28.626383 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:28.627090 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:28.627254 (+ 164us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:28.628565 (+ 1311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:28.628576 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:28.628590 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:17:28.629167 (+ 577us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:28.629243 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:17:28.629268 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:28.629460 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:28.629704 (+ 244us) server_negotiation.cc:299] Negotiation successful
0924 16:17:28.629898 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":408,"threads_started":1}
I20260924 16:17:28.631680 19118 log.cc:824] T 00000000000000000000000000000000 P 7199cfd144ed4859b9d91df4f435f89e: Log is configured to *not* fsync() on all Append() calls
I20260924 16:17:28.634764 19118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7199cfd144ed4859b9d91df4f435f89e: No bootstrap required, opened a new log
I20260924 16:17:28.635376 30632 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:17:28.645370 19118 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7199cfd144ed4859b9d91df4f435f89e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7199cfd144ed4859b9d91df4f435f89e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32849 } }
I20260924 16:17:28.645963 19118 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7199cfd144ed4859b9d91df4f435f89e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:17:28.646131 19118 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7199cfd144ed4859b9d91df4f435f89e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7199cfd144ed4859b9d91df4f435f89e, State: Initialized, Role: FOLLOWER
I20260924 16:17:28.646831 19118 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7199cfd144ed4859b9d91df4f435f89e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7199cfd144ed4859b9d91df4f435f89e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32849 } }
I20260924 16:17:28.647085 19118 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7199cfd144ed4859b9d91df4f435f89e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:17:28.647264 19118 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7199cfd144ed4859b9d91df4f435f89e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:17:28.647449 19118 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7199cfd144ed4859b9d91df4f435f89e [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:17:28.648181 19121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:28.640864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58461 (local address 127.0.0.1:32849)
0924 16:17:28.641084 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:28.641091 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:28.641550 (+ 459us) server_negotiation.cc:407] Connection header received
0924 16:17:28.641754 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:28.641760 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:28.641849 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:28.641951 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:28.642439 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:28.643626 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:28.645066 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:28.645733 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:28.645877 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:28.647258 (+ 1381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:28.647263 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:28.647266 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:17:28.647285 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:28.647349 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:17:28.647376 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:28.647505 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:28.647752 (+ 247us) server_negotiation.cc:299] Negotiation successful
0924 16:17:28.647812 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 16:17:28.650682 19118 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7199cfd144ed4859b9d91df4f435f89e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7199cfd144ed4859b9d91df4f435f89e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32849 } }
I20260924 16:17:28.651271 19118 leader_election.cc:302] T 00000000000000000000000000000000 P 7199cfd144ed4859b9d91df4f435f89e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7199cfd144ed4859b9d91df4f435f89e; no voters:
I20260924 16:17:28.651912 19118 leader_election.cc:288] T 00000000000000000000000000000000 P 7199cfd144ed4859b9d91df4f435f89e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:17:28.652062 19128 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7199cfd144ed4859b9d91df4f435f89e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:17:28.653535 19128 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7199cfd144ed4859b9d91df4f435f89e [term 1 LEADER]: Becoming Leader. State: Replica: 7199cfd144ed4859b9d91df4f435f89e, State: Running, Role: LEADER
I20260924 16:17:28.654069 19128 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7199cfd144ed4859b9d91df4f435f89e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7199cfd144ed4859b9d91df4f435f89e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32849 } }
I20260924 16:17:28.654559 19118 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7199cfd144ed4859b9d91df4f435f89e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:17:28.657269 19130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7199cfd144ed4859b9d91df4f435f89e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7199cfd144ed4859b9d91df4f435f89e. Latest consensus state: current_term: 1 leader_uuid: "7199cfd144ed4859b9d91df4f435f89e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7199cfd144ed4859b9d91df4f435f89e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32849 } } }
I20260924 16:17:28.657763 19130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7199cfd144ed4859b9d91df4f435f89e [sys.catalog]: This master's current role is: LEADER
I20260924 16:17:28.657258 19129 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7199cfd144ed4859b9d91df4f435f89e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7199cfd144ed4859b9d91df4f435f89e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7199cfd144ed4859b9d91df4f435f89e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32849 } } }
I20260924 16:17:28.658449 19129 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7199cfd144ed4859b9d91df4f435f89e [sys.catalog]: This master's current role is: LEADER
I20260924 16:17:28.663921 19134 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:17:28.668334 19134 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:17:28.676587 19134 catalog_manager.cc:1347] Generated new cluster ID: 7c8fcf5ca9b149feb64c020b70a47758
I20260924 16:17:28.676712 19134 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:17:28.690505 19134 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:17:28.691890 19134 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:17:28.700615 19134 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7199cfd144ed4859b9d91df4f435f89e: Generated new TSK 0
I20260924 16:17:28.701545 19134 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:17:28.713554 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 19050
W20260924 16:17:28.714833 19112 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:34381
W20260924 16:17:28.715641 30632 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 19050: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 16:17:28.715842 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 19050
W20260924 16:17:28.733789 19034 connection.cc:535] client connection to 127.0.0.1:32849 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32849): Connection reset by peer (error 104)
2026-09-24T16:17:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1961 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790266323169505-30632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:17:28 dist-test-slave-v5j9 krb5kdc[19149](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:17:28 dist-test-slave-v5j9 krb5kdc[19149](info): set up 2 sockets
Sep 24 16:17:28 dist-test-slave-v5j9 krb5kdc[19149](info): commencing operation
krb5kdc: starting...
W20260924 16:17:29.829250 30632 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.005s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Sep 24 16:17:30 dist-test-slave-v5j9 krb5kdc[19149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:17:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:17:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790266323169505-30632-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790266323169505-30632-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 16:17:30.109208 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790266323169505-30632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790266323169505-30632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790266323169505-30632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34653
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790266323169505-30632-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34653
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:17:30.448925 19163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:17:30.449267 19163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:17:30.449484 19163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:17:30.457280 19163 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:17:30.457477 19163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:17:30.457562 19163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:17:30.457674 19163 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:17:30.457753 19163 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:17:30.468101 19163 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790266323169505-30632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790266323169505-30632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34653
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790266323169505-30632-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790266323169505-30632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:17:30.469616 19163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:17:30.470969 19163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:17:30.483665 19168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:30.483735 19171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:30.483759 19169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:17:30.484407 19163 server_base.cc:1034] running on GCE node
I20260924 16:17:30.485328 19163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:17:30.486868 19163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:17:30.488067 19163 hybrid_clock.cc:648] HybridClock initialized: now 1790266650488005 us; error 51 us; skew 500 ppm
I20260924 16:17:30.488426 19163 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:17:30 dist-test-slave-v5j9 krb5kdc[19149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266650, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:17:30.491760 19163 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 16:17:30.493414 19163 webserver.cc:460] Webserver started at http://127.0.0.1:42453/ using document root <none> and password file <none>
I20260924 16:17:30.494174 19163 fs_manager.cc:359] Metadata directory not provided
I20260924 16:17:30.494349 19163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:17:30.494694 19163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:17:30.498885 19163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790266323169505-30632-0/minicluster-data/master-0/data/instance:
uuid: "a9a52e32553c4926a5110b28dc92cf7f"
format_stamp: "Formatted at 2026-09-24 16:17:30 on dist-test-slave-v5j9"
server_key: "747ecb91625507f1afefa3e71da7d047"
server_key_iv: "a251175eb3c10d497f53789f3f8c1ff4"
server_key_version: "encryptionkey@0"
I20260924 16:17:30.499779 19163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790266323169505-30632-0/minicluster-data/master-0/wal/instance:
uuid: "a9a52e32553c4926a5110b28dc92cf7f"
format_stamp: "Formatted at 2026-09-24 16:17:30 on dist-test-slave-v5j9"
server_key: "747ecb91625507f1afefa3e71da7d047"
server_key_iv: "a251175eb3c10d497f53789f3f8c1ff4"
server_key_version: "encryptionkey@0"
I20260924 16:17:30.505601 19163 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260924 16:17:30.509310 19178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:17:30.510838 19163 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:17:30.511088 19163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790266323169505-30632-0/minicluster-data/master-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790266323169505-30632-0/minicluster-data/master-0/wal
uuid: "a9a52e32553c4926a5110b28dc92cf7f"
format_stamp: "Formatted at 2026-09-24 16:17:30 on dist-test-slave-v5j9"
server_key: "747ecb91625507f1afefa3e71da7d047"
server_key_iv: "a251175eb3c10d497f53789f3f8c1ff4"
server_key_version: "encryptionkey@0"
I20260924 16:17:30.511354 19163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790266323169505-30632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790266323169505-30632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790266323169505-30632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:17:30.517575 19163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:17:30.521845 19163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:17:30.522177 19163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:17:30.541842 19163 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34653
I20260924 16:17:30.541884 19230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34653 every 8 connection(s)
I20260924 16:17:30.543408 19163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
I20260924 16:17:30.545596 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 19163
I20260924 16:17:30.545842 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790266323169505-30632-0/minicluster-data/master-0/wal/instance
I20260924 16:17:30.546411 30632 external_mini_cluster.cc:1428] Setting key 5e54e1bb487f2ddb85c589cd378dfa6d
I20260924 16:17:30.548328 19231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:17:30.555595 19234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:30.548296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47977 (local address 127.0.0.1:34653)
0924 16:17:30.549124 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:30.549138 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:30.549189 (+ 51us) server_negotiation.cc:407] Connection header received
0924 16:17:30.551393 (+ 2204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:30.551420 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:30.551909 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:30.552377 (+ 468us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:30.553413 (+ 1036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:30.553423 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:30.553450 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:17:30.553470 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:30.553563 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:17:30.553596 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:30.553844 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:30.553983 (+ 139us) server_negotiation.cc:299] Negotiation successful
0924 16:17:30.554226 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":326,"threads_started":1}
I20260924 16:17:30.559643 30632 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:17:30.562657 19231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9a52e32553c4926a5110b28dc92cf7f: Bootstrap starting.
I20260924 16:17:30.566756 19231 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9a52e32553c4926a5110b28dc92cf7f: Neither blocks nor log segments found. Creating new log.
I20260924 16:17:30.567432 19234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:30.564734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50741 (local address 127.0.0.1:34653)
0924 16:17:30.564958 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:30.564966 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:30.565291 (+ 325us) server_negotiation.cc:407] Connection header received
0924 16:17:30.565493 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:30.565496 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:30.565547 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:30.565628 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:30.566673 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:30.566677 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:30.566680 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:17:30.566694 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:30.566734 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:17:30.566752 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:30.566841 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:30.567052 (+ 211us) server_negotiation.cc:299] Negotiation successful
0924 16:17:30.567103 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260924 16:17:30.568127 19231 log.cc:824] T 00000000000000000000000000000000 P a9a52e32553c4926a5110b28dc92cf7f: Log is configured to *not* fsync() on all Append() calls
I20260924 16:17:30.570977 19231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9a52e32553c4926a5110b28dc92cf7f: No bootstrap required, opened a new log
I20260924 16:17:30.578444 19231 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9a52e32553c4926a5110b28dc92cf7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9a52e32553c4926a5110b28dc92cf7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34653 } }
I20260924 16:17:30.578863 19231 raft_consensus.cc:374] T 00000000000000000000000000000000 P a9a52e32553c4926a5110b28dc92cf7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:17:30.578991 19231 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9a52e32553c4926a5110b28dc92cf7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9a52e32553c4926a5110b28dc92cf7f, State: Initialized, Role: FOLLOWER
I20260924 16:17:30.579607 19231 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9a52e32553c4926a5110b28dc92cf7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9a52e32553c4926a5110b28dc92cf7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34653 } }
I20260924 16:17:30.579841 19231 raft_consensus.cc:388] T 00000000000000000000000000000000 P a9a52e32553c4926a5110b28dc92cf7f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:17:30.580008 19231 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9a52e32553c4926a5110b28dc92cf7f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:17:30.580194 19231 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9a52e32553c4926a5110b28dc92cf7f [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:17:30.582572 19231 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9a52e32553c4926a5110b28dc92cf7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9a52e32553c4926a5110b28dc92cf7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34653 } }
I20260924 16:17:30.583117 19231 leader_election.cc:302] T 00000000000000000000000000000000 P a9a52e32553c4926a5110b28dc92cf7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9a52e32553c4926a5110b28dc92cf7f; no voters:
I20260924 16:17:30.583745 19231 leader_election.cc:288] T 00000000000000000000000000000000 P a9a52e32553c4926a5110b28dc92cf7f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:17:30.583921 19241 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9a52e32553c4926a5110b28dc92cf7f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:17:30.585163 19241 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9a52e32553c4926a5110b28dc92cf7f [term 1 LEADER]: Becoming Leader. State: Replica: a9a52e32553c4926a5110b28dc92cf7f, State: Running, Role: LEADER
I20260924 16:17:30.585879 19241 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9a52e32553c4926a5110b28dc92cf7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9a52e32553c4926a5110b28dc92cf7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34653 } }
I20260924 16:17:30.586181 19231 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9a52e32553c4926a5110b28dc92cf7f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:17:30.588858 19243 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9a52e32553c4926a5110b28dc92cf7f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9a52e32553c4926a5110b28dc92cf7f. Latest consensus state: current_term: 1 leader_uuid: "a9a52e32553c4926a5110b28dc92cf7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9a52e32553c4926a5110b28dc92cf7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34653 } } }
I20260924 16:17:30.589046 19242 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9a52e32553c4926a5110b28dc92cf7f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a9a52e32553c4926a5110b28dc92cf7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9a52e32553c4926a5110b28dc92cf7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34653 } } }
I20260924 16:17:30.589434 19243 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9a52e32553c4926a5110b28dc92cf7f [sys.catalog]: This master's current role is: LEADER
I20260924 16:17:30.589447 19242 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9a52e32553c4926a5110b28dc92cf7f [sys.catalog]: This master's current role is: LEADER
I20260924 16:17:30.590694 19248 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:17:30.596431 19248 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:17:30.605818 19248 catalog_manager.cc:1347] Generated new cluster ID: 946293434e8d48968e8b544b64e1e2a2
I20260924 16:17:30.605949 19248 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:17:30.624536 19248 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:17:30.625010 19248 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:17:30.634042 19248 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a9a52e32553c4926a5110b28dc92cf7f: Generated new TSK 0
I20260924 16:17:30.634925 19248 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:17:30.647773 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 19163
W20260924 16:17:30.649055 19226 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:47977
W20260924 16:17:30.649897 30632 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 19163: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 16:17:30.650064 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 19163
2026-09-24T16:17:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1934 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790266323169505-30632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:17:30 dist-test-slave-v5j9 krb5kdc[19263](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:17:30 dist-test-slave-v5j9 krb5kdc[19263](info): set up 2 sockets
Sep 24 16:17:30 dist-test-slave-v5j9 krb5kdc[19263](info): commencing operation
krb5kdc: starting...
W20260924 16:17:31.769457 30632 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Sep 24 16:17:31 dist-test-slave-v5j9 krb5kdc[19263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:17:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:17:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790266323169505-30632-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790266323169505-30632-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 16:17:32.065248 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790266323169505-30632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790266323169505-30632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790266323169505-30632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42773
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790266323169505-30632-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42773
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:17:32.419485 19276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:17:32.419826 19276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:17:32.420008 19276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:17:32.427446 19276 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:17:32.427575 19276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:17:32.427707 19276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:17:32.427800 19276 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:17:32.427870 19276 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:17:32.438110 19276 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790266323169505-30632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790266323169505-30632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42773
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790266323169505-30632-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790266323169505-30632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:17:32.439538 19276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:17:32.440982 19276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:17:32.453424 19284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:32.453465 19281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:32.453500 19282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:17:32.454316 19276 server_base.cc:1034] running on GCE node
I20260924 16:17:32.455180 19276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:17:32.456707 19276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:17:32.457943 19276 hybrid_clock.cc:648] HybridClock initialized: now 1790266652457893 us; error 38 us; skew 500 ppm
I20260924 16:17:32.458330 19276 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:17:32 dist-test-slave-v5j9 krb5kdc[19263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:17:32.461714 19276 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 16:17:32.463332 19276 webserver.cc:460] Webserver started at http://127.0.0.1:43581/ using document root <none> and password file <none>
I20260924 16:17:32.464087 19276 fs_manager.cc:359] Metadata directory not provided
I20260924 16:17:32.464246 19276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:17:32.464566 19276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:17:32.467892 19276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790266323169505-30632-0/minicluster-data/master-0/data/instance:
uuid: "6ca564a096314fa7aec0eb9831b16e3b"
format_stamp: "Formatted at 2026-09-24 16:17:32 on dist-test-slave-v5j9"
server_key: "6d2c718a7adc870b77c6a34757faaa87"
server_key_iv: "b25246c372686d88baf25d0a0993ee05"
server_key_version: "encryptionkey@0"
I20260924 16:17:32.468729 19276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790266323169505-30632-0/minicluster-data/master-0/wal/instance:
uuid: "6ca564a096314fa7aec0eb9831b16e3b"
format_stamp: "Formatted at 2026-09-24 16:17:32 on dist-test-slave-v5j9"
server_key: "6d2c718a7adc870b77c6a34757faaa87"
server_key_iv: "b25246c372686d88baf25d0a0993ee05"
server_key_version: "encryptionkey@0"
I20260924 16:17:32.474543 19276 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260924 16:17:32.478310 19291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:17:32.479771 19276 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:17:32.479970 19276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790266323169505-30632-0/minicluster-data/master-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790266323169505-30632-0/minicluster-data/master-0/wal
uuid: "6ca564a096314fa7aec0eb9831b16e3b"
format_stamp: "Formatted at 2026-09-24 16:17:32 on dist-test-slave-v5j9"
server_key: "6d2c718a7adc870b77c6a34757faaa87"
server_key_iv: "b25246c372686d88baf25d0a0993ee05"
server_key_version: "encryptionkey@0"
I20260924 16:17:32.480181 19276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790266323169505-30632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790266323169505-30632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790266323169505-30632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:17:32.492650 19276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:17:32.497103 19276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:17:32.497496 19276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:17:32.517889 19276 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42773
I20260924 16:17:32.517918 19345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42773 every 8 connection(s)
I20260924 16:17:32.519485 19276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
I20260924 16:17:32.522215 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 19276
I20260924 16:17:32.522440 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790266323169505-30632-0/minicluster-data/master-0/wal/instance
I20260924 16:17:32.522892 30632 external_mini_cluster.cc:1428] Setting key 47065ba050f6ad215dec896d7dd080ad
I20260924 16:17:32.523994 19346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 16:17:32 dist-test-slave-v5j9 krb5kdc[19263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 16:17:32.538982 19346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ca564a096314fa7aec0eb9831b16e3b: Bootstrap starting.
I20260924 16:17:32.543537 19346 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ca564a096314fa7aec0eb9831b16e3b: Neither blocks nor log segments found. Creating new log.
I20260924 16:17:32.544842 19346 log.cc:824] T 00000000000000000000000000000000 P 6ca564a096314fa7aec0eb9831b16e3b: Log is configured to *not* fsync() on all Append() calls
I20260924 16:17:32.548630 19346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ca564a096314fa7aec0eb9831b16e3b: No bootstrap required, opened a new log
I20260924 16:17:32.548687 19349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:32.524837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38359 (local address 127.0.0.1:42773)
0924 16:17:32.525760 (+ 923us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:32.525779 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:32.525845 (+ 66us) server_negotiation.cc:407] Connection header received
0924 16:17:32.527809 (+ 1964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:32.527843 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:32.528446 (+ 603us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:32.528968 (+ 522us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:32.530445 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:32.532547 (+ 2102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:32.533883 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:32.534980 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:32.535171 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:32.539561 (+ 4390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:32.539573 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:32.539595 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:17:32.539648 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:32.543729 (+ 4081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:17:32.544527 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:17:32.544535 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:17:32.544546 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:17:32.544766 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:17:32.545213 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:17:32.545218 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:17:32.545222 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:17:32.545971 (+ 749us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:32.546223 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:32.546820 (+ 597us) server_negotiation.cc:299] Negotiation successful
0924 16:17:32.547282 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":451,"threads_started":1}
I20260924 16:17:32.552917 30632 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:17:32.557264 19346 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ca564a096314fa7aec0eb9831b16e3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ca564a096314fa7aec0eb9831b16e3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42773 } }
I20260924 16:17:32.557898 19346 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ca564a096314fa7aec0eb9831b16e3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:17:32.558030 19346 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ca564a096314fa7aec0eb9831b16e3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ca564a096314fa7aec0eb9831b16e3b, State: Initialized, Role: FOLLOWER
I20260924 16:17:32.558784 19346 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ca564a096314fa7aec0eb9831b16e3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ca564a096314fa7aec0eb9831b16e3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42773 } }
I20260924 16:17:32.559098 19346 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ca564a096314fa7aec0eb9831b16e3b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:17:32.559252 19346 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ca564a096314fa7aec0eb9831b16e3b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:17:32.559465 19346 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ca564a096314fa7aec0eb9831b16e3b [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:17:32.562597 19346 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ca564a096314fa7aec0eb9831b16e3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ca564a096314fa7aec0eb9831b16e3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42773 } }
I20260924 16:17:32.563243 19346 leader_election.cc:302] T 00000000000000000000000000000000 P 6ca564a096314fa7aec0eb9831b16e3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ca564a096314fa7aec0eb9831b16e3b; no voters:
I20260924 16:17:32.564132 19346 leader_election.cc:288] T 00000000000000000000000000000000 P 6ca564a096314fa7aec0eb9831b16e3b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:17:32.564257 19356 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ca564a096314fa7aec0eb9831b16e3b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:17:32.567024 19346 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ca564a096314fa7aec0eb9831b16e3b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:17:32.568499 19356 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ca564a096314fa7aec0eb9831b16e3b [term 1 LEADER]: Becoming Leader. State: Replica: 6ca564a096314fa7aec0eb9831b16e3b, State: Running, Role: LEADER
I20260924 16:17:32.569522 19356 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ca564a096314fa7aec0eb9831b16e3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ca564a096314fa7aec0eb9831b16e3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42773 } }
I20260924 16:17:32.573071 19358 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ca564a096314fa7aec0eb9831b16e3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ca564a096314fa7aec0eb9831b16e3b. Latest consensus state: current_term: 1 leader_uuid: "6ca564a096314fa7aec0eb9831b16e3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ca564a096314fa7aec0eb9831b16e3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42773 } } }
I20260924 16:17:32.573570 19358 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ca564a096314fa7aec0eb9831b16e3b [sys.catalog]: This master's current role is: LEADER
I20260924 16:17:32.574219 19357 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ca564a096314fa7aec0eb9831b16e3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ca564a096314fa7aec0eb9831b16e3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ca564a096314fa7aec0eb9831b16e3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42773 } } }
I20260924 16:17:32.574651 19357 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ca564a096314fa7aec0eb9831b16e3b [sys.catalog]: This master's current role is: LEADER
I20260924 16:17:32.575088 19349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:32.557502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55745 (local address 127.0.0.1:42773)
0924 16:17:32.557733 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:32.557743 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:32.558233 (+ 490us) server_negotiation.cc:407] Connection header received
0924 16:17:32.558460 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:32.558468 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:32.558554 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:32.558638 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:32.560857 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:32.561739 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:32.563539 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:32.564188 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:32.565122 (+ 934us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:32.567343 (+ 2221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:32.567348 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:32.567351 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:17:32.567385 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:32.570894 (+ 3509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:17:32.571785 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:17:32.571796 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:17:32.571799 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:17:32.571983 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:17:32.572507 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:17:32.572514 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:17:32.572520 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:17:32.572916 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:32.573220 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:32.574486 (+ 1266us) server_negotiation.cc:299] Negotiation successful
0924 16:17:32.574731 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 16:17:32.583595 19363 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:17:32.589291 19363 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:17:32.599709 19363 catalog_manager.cc:1347] Generated new cluster ID: 1e581d966b1e40c98f29c3d7cc1a3a1e
I20260924 16:17:32.599856 19363 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:17:32.635571 19363 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:17:32.637022 19363 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:17:32.649273 19363 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ca564a096314fa7aec0eb9831b16e3b: Generated new TSK 0
I20260924 16:17:32.650316 19363 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:17:32.664541 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 19276
I20260924 16:17:32.666244 19340 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:17:32.763355 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 19276
2026-09-24T16:17:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2113 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790266323169505-30632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:17:32 dist-test-slave-v5j9 krb5kdc[19379](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:17:32 dist-test-slave-v5j9 krb5kdc[19379](info): set up 2 sockets
Sep 24 16:17:32 dist-test-slave-v5j9 krb5kdc[19379](info): commencing operation
krb5kdc: starting...
W20260924 16:17:33.881646 30632 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.005s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Sep 24 16:17:34 dist-test-slave-v5j9 krb5kdc[19379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:17:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:17:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790266323169505-30632-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790266323169505-30632-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 16:17:34.144097 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790266323169505-30632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790266323169505-30632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790266323169505-30632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35665
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790266323169505-30632-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35665
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:17:34.498265 19392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:17:34.498631 19392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:17:34.498793 19392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:17:34.506297 19392 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:17:34.506456 19392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:17:34.506567 19392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:17:34.506673 19392 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:17:34.506759 19392 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:17:34.517940 19392 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790266323169505-30632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790266323169505-30632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35665
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790266323169505-30632-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35665
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790266323169505-30632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:17:34.519308 19392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:17:34.520630 19392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:17:34.532604 19397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:34.532641 19400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:34.532567 19398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:17:34.533730 19392 server_base.cc:1034] running on GCE node
I20260924 16:17:34.534535 19392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:17:34.535953 19392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:17:34.537189 19392 hybrid_clock.cc:648] HybridClock initialized: now 1790266654537142 us; error 59 us; skew 500 ppm
I20260924 16:17:34.537585 19392 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:17:34 dist-test-slave-v5j9 krb5kdc[19379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266654, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:17:34.546320 19392 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 16:17:34.547912 19392 webserver.cc:460] Webserver started at http://127.0.0.1:38495/ using document root <none> and password file <none>
I20260924 16:17:34.548681 19392 fs_manager.cc:359] Metadata directory not provided
I20260924 16:17:34.548833 19392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:17:34.549165 19392 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:17:34.552457 19392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790266323169505-30632-0/minicluster-data/master-0/data/instance:
uuid: "be3a89d7930b467e95535cc299d727ae"
format_stamp: "Formatted at 2026-09-24 16:17:34 on dist-test-slave-v5j9"
server_key: "9d9d4c8d4ac040540ebe27d592bc5da3"
server_key_iv: "bce3ee2f7aa4dda9980103a24117a00b"
server_key_version: "encryptionkey@0"
I20260924 16:17:34.553289 19392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790266323169505-30632-0/minicluster-data/master-0/wal/instance:
uuid: "be3a89d7930b467e95535cc299d727ae"
format_stamp: "Formatted at 2026-09-24 16:17:34 on dist-test-slave-v5j9"
server_key: "9d9d4c8d4ac040540ebe27d592bc5da3"
server_key_iv: "bce3ee2f7aa4dda9980103a24117a00b"
server_key_version: "encryptionkey@0"
I20260924 16:17:34.558815 19392 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260924 16:17:34.562465 19407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:17:34.563868 19392 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 16:17:34.564070 19392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790266323169505-30632-0/minicluster-data/master-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790266323169505-30632-0/minicluster-data/master-0/wal
uuid: "be3a89d7930b467e95535cc299d727ae"
format_stamp: "Formatted at 2026-09-24 16:17:34 on dist-test-slave-v5j9"
server_key: "9d9d4c8d4ac040540ebe27d592bc5da3"
server_key_iv: "bce3ee2f7aa4dda9980103a24117a00b"
server_key_version: "encryptionkey@0"
I20260924 16:17:34.564282 19392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790266323169505-30632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790266323169505-30632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790266323169505-30632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:17:34.587054 19392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:17:34.591782 19392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:17:34.592095 19392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:17:34.610903 19392 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35665
I20260924 16:17:34.610940 19459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35665 every 8 connection(s)
I20260924 16:17:34.612339 19392 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
I20260924 16:17:34.616246 19460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:17:34.620836 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 19392
I20260924 16:17:34.621016 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790266323169505-30632-0/minicluster-data/master-0/wal/instance
I20260924 16:17:34.621620 30632 external_mini_cluster.cc:1428] Setting key b7b766a760ea6a7e24940dffb8967789
I20260924 16:17:34.628057 19460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be3a89d7930b467e95535cc299d727ae: Bootstrap starting.
I20260924 16:17:34.631773 19460 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be3a89d7930b467e95535cc299d727ae: Neither blocks nor log segments found. Creating new log.
I20260924 16:17:34.633123 19460 log.cc:824] T 00000000000000000000000000000000 P be3a89d7930b467e95535cc299d727ae: Log is configured to *not* fsync() on all Append() calls
I20260924 16:17:34.636098 19463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:34.623585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59007 (local address 127.0.0.1:35665)
0924 16:17:34.624412 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:34.624427 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:34.624485 (+ 58us) server_negotiation.cc:407] Connection header received
0924 16:17:34.626525 (+ 2040us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:34.626553 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:34.626991 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:34.627337 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:34.627977 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:34.629416 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:34.631327 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:34.632088 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:34.632240 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:34.633461 (+ 1221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:34.633470 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:34.633484 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:17:34.634329 (+ 845us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:34.634434 (+ 105us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:17:34.634468 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:34.634603 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:34.634836 (+ 233us) server_negotiation.cc:299] Negotiation successful
0924 16:17:34.635153 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":384,"threads_started":1}
I20260924 16:17:34.636264 19460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be3a89d7930b467e95535cc299d727ae: No bootstrap required, opened a new log
I20260924 16:17:34.639773 30632 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:17:34.646103 19460 raft_consensus.cc:348] T 00000000000000000000000000000000 P be3a89d7930b467e95535cc299d727ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be3a89d7930b467e95535cc299d727ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35665 } }
I20260924 16:17:34.646554 19460 raft_consensus.cc:374] T 00000000000000000000000000000000 P be3a89d7930b467e95535cc299d727ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:17:34.646677 19460 raft_consensus.cc:729] T 00000000000000000000000000000000 P be3a89d7930b467e95535cc299d727ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be3a89d7930b467e95535cc299d727ae, State: Initialized, Role: FOLLOWER
I20260924 16:17:34.647367 19460 consensus_queue.cc:260] T 00000000000000000000000000000000 P be3a89d7930b467e95535cc299d727ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be3a89d7930b467e95535cc299d727ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35665 } }
I20260924 16:17:34.647665 19460 raft_consensus.cc:388] T 00000000000000000000000000000000 P be3a89d7930b467e95535cc299d727ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:17:34.647847 19460 raft_consensus.cc:482] T 00000000000000000000000000000000 P be3a89d7930b467e95535cc299d727ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:17:34.648015 19460 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be3a89d7930b467e95535cc299d727ae [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:17:34.650794 19460 raft_consensus.cc:504] T 00000000000000000000000000000000 P be3a89d7930b467e95535cc299d727ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be3a89d7930b467e95535cc299d727ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35665 } }
I20260924 16:17:34.651464 19460 leader_election.cc:302] T 00000000000000000000000000000000 P be3a89d7930b467e95535cc299d727ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be3a89d7930b467e95535cc299d727ae; no voters:
I20260924 16:17:34.651748 19463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:34.644899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42753 (local address 127.0.0.1:35665)
0924 16:17:34.645095 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:34.645101 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:34.645631 (+ 530us) server_negotiation.cc:407] Connection header received
0924 16:17:34.645793 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:34.645798 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:34.645880 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:34.645984 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:34.646470 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:34.647527 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:34.649015 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:34.649758 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:34.649897 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:34.650967 (+ 1070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:34.650974 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:34.650978 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:17:34.651000 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:34.651046 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:17:34.651065 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:34.651161 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:34.651305 (+ 144us) server_negotiation.cc:299] Negotiation successful
0924 16:17:34.651395 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 16:17:34.652231 19460 leader_election.cc:288] T 00000000000000000000000000000000 P be3a89d7930b467e95535cc299d727ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:17:34.652360 19470 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be3a89d7930b467e95535cc299d727ae [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:17:34.653626 19470 raft_consensus.cc:686] T 00000000000000000000000000000000 P be3a89d7930b467e95535cc299d727ae [term 1 LEADER]: Becoming Leader. State: Replica: be3a89d7930b467e95535cc299d727ae, State: Running, Role: LEADER
I20260924 16:17:34.654225 19470 consensus_queue.cc:237] T 00000000000000000000000000000000 P be3a89d7930b467e95535cc299d727ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be3a89d7930b467e95535cc299d727ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35665 } }
I20260924 16:17:34.654824 19460 sys_catalog.cc:564] T 00000000000000000000000000000000 P be3a89d7930b467e95535cc299d727ae [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:17:34.657392 19471 sys_catalog.cc:455] T 00000000000000000000000000000000 P be3a89d7930b467e95535cc299d727ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be3a89d7930b467e95535cc299d727ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be3a89d7930b467e95535cc299d727ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35665 } } }
I20260924 16:17:34.657809 19471 sys_catalog.cc:458] T 00000000000000000000000000000000 P be3a89d7930b467e95535cc299d727ae [sys.catalog]: This master's current role is: LEADER
I20260924 16:17:34.658566 19472 sys_catalog.cc:455] T 00000000000000000000000000000000 P be3a89d7930b467e95535cc299d727ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader be3a89d7930b467e95535cc299d727ae. Latest consensus state: current_term: 1 leader_uuid: "be3a89d7930b467e95535cc299d727ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be3a89d7930b467e95535cc299d727ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35665 } } }
I20260924 16:17:34.658872 19476 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:17:34.659823 19472 sys_catalog.cc:458] T 00000000000000000000000000000000 P be3a89d7930b467e95535cc299d727ae [sys.catalog]: This master's current role is: LEADER
I20260924 16:17:34.666858 19476 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:17:34.676220 19476 catalog_manager.cc:1347] Generated new cluster ID: 33688c2c48824c95a33daaae2834b2a7
I20260924 16:17:34.676347 19476 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:17:34.718187 19476 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:17:34.719657 19476 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:17:34.730974 19476 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be3a89d7930b467e95535cc299d727ae: Generated new TSK 0
I20260924 16:17:34.731917 19476 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:17:34.743945 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 19392
W20260924 16:17:34.745157 19455 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:59007
W20260924 16:17:34.745934 30632 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 19392: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 16:17:34.746104 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 19392
2026-09-24T16:17:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1982 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790266323169505-30632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:17:34 dist-test-slave-v5j9 krb5kdc[19492](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:17:34 dist-test-slave-v5j9 krb5kdc[19492](info): set up 2 sockets
Sep 24 16:17:34 dist-test-slave-v5j9 krb5kdc[19492](info): commencing operation
krb5kdc: starting...
W20260924 16:17:35.874222 30632 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.003s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Sep 24 16:17:36 dist-test-slave-v5j9 krb5kdc[19492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:17:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:17:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790266323169505-30632-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790266323169505-30632-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 16:17:36.150580 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790266323169505-30632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790266323169505-30632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790266323169505-30632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33851
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790266323169505-30632-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33851
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:17:36.494843 19505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:17:36.495193 19505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:17:36.495373 19505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:17:36.502868 19505 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:17:36.502990 19505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:17:36.503036 19505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:17:36.503204 19505 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:17:36.503268 19505 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:17:36.513492 19505 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790266323169505-30632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790266323169505-30632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33851
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790266323169505-30632-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790266323169505-30632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:17:36.514878 19505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:17:36.516243 19505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:17:36.529520 19510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:36.529539 19511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:36.529565 19513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:17:36.530431 19505 server_base.cc:1034] running on GCE node
I20260924 16:17:36.531214 19505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:17:36.532606 19505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:17:36.533849 19505 hybrid_clock.cc:648] HybridClock initialized: now 1790266656533788 us; error 75 us; skew 500 ppm
I20260924 16:17:36.534221 19505 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:17:36 dist-test-slave-v5j9 krb5kdc[19492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266656, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:17:36.537321 19505 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 16:17:36.538918 19505 webserver.cc:460] Webserver started at http://127.0.0.1:42907/ using document root <none> and password file <none>
I20260924 16:17:36.539619 19505 fs_manager.cc:359] Metadata directory not provided
I20260924 16:17:36.539765 19505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:17:36.540091 19505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:17:36.543155 19505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790266323169505-30632-0/minicluster-data/master-0/data/instance:
uuid: "40108faef48d4e89b17d2bbdfcf6a267"
format_stamp: "Formatted at 2026-09-24 16:17:36 on dist-test-slave-v5j9"
server_key: "72583c10032c04e20cd57ab597f5f497"
server_key_iv: "0371b710ac4c8b1b283586dfdeacf5de"
server_key_version: "encryptionkey@0"
I20260924 16:17:36.543942 19505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790266323169505-30632-0/minicluster-data/master-0/wal/instance:
uuid: "40108faef48d4e89b17d2bbdfcf6a267"
format_stamp: "Formatted at 2026-09-24 16:17:36 on dist-test-slave-v5j9"
server_key: "72583c10032c04e20cd57ab597f5f497"
server_key_iv: "0371b710ac4c8b1b283586dfdeacf5de"
server_key_version: "encryptionkey@0"
I20260924 16:17:36.549239 19505 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 16:17:36.552727 19520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:17:36.554284 19505 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:17:36.554522 19505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790266323169505-30632-0/minicluster-data/master-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790266323169505-30632-0/minicluster-data/master-0/wal
uuid: "40108faef48d4e89b17d2bbdfcf6a267"
format_stamp: "Formatted at 2026-09-24 16:17:36 on dist-test-slave-v5j9"
server_key: "72583c10032c04e20cd57ab597f5f497"
server_key_iv: "0371b710ac4c8b1b283586dfdeacf5de"
server_key_version: "encryptionkey@0"
I20260924 16:17:36.554749 19505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790266323169505-30632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790266323169505-30632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790266323169505-30632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:17:36.560809 19505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:17:36.566201 19505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:17:36.566551 19505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:17:36.587127 19505 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33851
I20260924 16:17:36.587241 19572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33851 every 8 connection(s)
I20260924 16:17:36.589020 19505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
I20260924 16:17:36.592981 19573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:17:36.595613 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 19505
I20260924 16:17:36.595808 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790266323169505-30632-0/minicluster-data/master-0/wal/instance
I20260924 16:17:36.596242 30632 external_mini_cluster.cc:1428] Setting key 5872163a29062ec826ff509fbddfdebd
I20260924 16:17:36.605228 19577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:36.598189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35267 (local address 127.0.0.1:33851)
0924 16:17:36.599014 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:36.599029 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:36.599079 (+ 50us) server_negotiation.cc:407] Connection header received
0924 16:17:36.601329 (+ 2250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:36.601398 (+ 69us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:36.601854 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:36.602303 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:36.603274 (+ 971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:36.603315 (+ 41us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:36.603333 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:17:36.603357 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:36.603429 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:17:36.603469 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:36.603684 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:36.603844 (+ 160us) server_negotiation.cc:299] Negotiation successful
0924 16:17:36.604043 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":327,"threads_started":1}
I20260924 16:17:36.606173 19573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40108faef48d4e89b17d2bbdfcf6a267: Bootstrap starting.
I20260924 16:17:36.608953 30632 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:17:36.609433 19573 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40108faef48d4e89b17d2bbdfcf6a267: Neither blocks nor log segments found. Creating new log.
I20260924 16:17:36.610409 19573 log.cc:824] T 00000000000000000000000000000000 P 40108faef48d4e89b17d2bbdfcf6a267: Log is configured to *not* fsync() on all Append() calls
I20260924 16:17:36.613179 19573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40108faef48d4e89b17d2bbdfcf6a267: No bootstrap required, opened a new log
I20260924 16:17:36.616106 19577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:36.613516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47853 (local address 127.0.0.1:33851)
0924 16:17:36.613755 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:36.613764 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:36.614187 (+ 423us) server_negotiation.cc:407] Connection header received
0924 16:17:36.614352 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:36.614356 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:36.614401 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:36.614481 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:36.615430 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:36.615434 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:36.615437 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:17:36.615454 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:36.615491 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:17:36.615507 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:36.615615 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:36.615726 (+ 111us) server_negotiation.cc:299] Negotiation successful
0924 16:17:36.615776 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 16:17:36.621670 19573 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40108faef48d4e89b17d2bbdfcf6a267 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40108faef48d4e89b17d2bbdfcf6a267" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33851 } }
I20260924 16:17:36.622083 19573 raft_consensus.cc:374] T 00000000000000000000000000000000 P 40108faef48d4e89b17d2bbdfcf6a267 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:17:36.622191 19573 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40108faef48d4e89b17d2bbdfcf6a267 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40108faef48d4e89b17d2bbdfcf6a267, State: Initialized, Role: FOLLOWER
I20260924 16:17:36.622763 19573 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40108faef48d4e89b17d2bbdfcf6a267 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40108faef48d4e89b17d2bbdfcf6a267" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33851 } }
I20260924 16:17:36.623020 19573 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40108faef48d4e89b17d2bbdfcf6a267 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:17:36.623172 19573 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40108faef48d4e89b17d2bbdfcf6a267 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:17:36.623345 19573 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40108faef48d4e89b17d2bbdfcf6a267 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:17:36.625602 19573 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40108faef48d4e89b17d2bbdfcf6a267 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40108faef48d4e89b17d2bbdfcf6a267" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33851 } }
I20260924 16:17:36.626153 19573 leader_election.cc:302] T 00000000000000000000000000000000 P 40108faef48d4e89b17d2bbdfcf6a267 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40108faef48d4e89b17d2bbdfcf6a267; no voters:
I20260924 16:17:36.626818 19573 leader_election.cc:288] T 00000000000000000000000000000000 P 40108faef48d4e89b17d2bbdfcf6a267 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:17:36.626940 19584 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40108faef48d4e89b17d2bbdfcf6a267 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:17:36.627637 19584 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40108faef48d4e89b17d2bbdfcf6a267 [term 1 LEADER]: Becoming Leader. State: Replica: 40108faef48d4e89b17d2bbdfcf6a267, State: Running, Role: LEADER
I20260924 16:17:36.628242 19584 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40108faef48d4e89b17d2bbdfcf6a267 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40108faef48d4e89b17d2bbdfcf6a267" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33851 } }
I20260924 16:17:36.629236 19573 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40108faef48d4e89b17d2bbdfcf6a267 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:17:36.631251 19586 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40108faef48d4e89b17d2bbdfcf6a267 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40108faef48d4e89b17d2bbdfcf6a267. Latest consensus state: current_term: 1 leader_uuid: "40108faef48d4e89b17d2bbdfcf6a267" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40108faef48d4e89b17d2bbdfcf6a267" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33851 } } }
I20260924 16:17:36.631752 19586 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40108faef48d4e89b17d2bbdfcf6a267 [sys.catalog]: This master's current role is: LEADER
I20260924 16:17:36.632175 19585 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40108faef48d4e89b17d2bbdfcf6a267 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "40108faef48d4e89b17d2bbdfcf6a267" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40108faef48d4e89b17d2bbdfcf6a267" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33851 } } }
I20260924 16:17:36.632463 19585 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40108faef48d4e89b17d2bbdfcf6a267 [sys.catalog]: This master's current role is: LEADER
I20260924 16:17:36.633876 19590 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:17:36.638556 19590 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:17:36.648444 19590 catalog_manager.cc:1347] Generated new cluster ID: d7b55bab70544611800575b49bdceae2
I20260924 16:17:36.648577 19590 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:17:36.658929 19590 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:17:36.659370 19590 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:17:36.668102 19590 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 40108faef48d4e89b17d2bbdfcf6a267: Generated new TSK 0
I20260924 16:17:36.669189 19590 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:17:36.682212 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 19505
W20260924 16:17:36.683542 19568 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:35267
W20260924 16:17:36.684338 30632 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 19505: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 16:17:36.684496 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 19505
2026-09-24T16:17:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1938 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790266323169505-30632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:17:36 dist-test-slave-v5j9 krb5kdc[19605](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:17:36 dist-test-slave-v5j9 krb5kdc[19605](info): set up 2 sockets
Sep 24 16:17:36 dist-test-slave-v5j9 krb5kdc[19605](info): commencing operation
krb5kdc: starting...
W20260924 16:17:37.829746 30632 mini_kdc.cc:121] Time spent starting KDC: real 1.108s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Sep 24 16:17:37 dist-test-slave-v5j9 krb5kdc[19605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266657, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:17:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:17:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790266323169505-30632-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790266323169505-30632-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 16:17:38.102495 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790266323169505-30632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790266323169505-30632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790266323169505-30632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35613
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790266323169505-30632-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35613
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:17:38.434548 19619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:17:38.434906 19619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:17:38.435066 19619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:17:38.441885 19619 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:17:38.442034 19619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:17:38.442134 19619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:17:38.442236 19619 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:17:38.442315 19619 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:17:38.451884 19619 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790266323169505-30632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790266323169505-30632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35613
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790266323169505-30632-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790266323169505-30632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:17:38.453351 19619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:17:38.454896 19619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:17:38.467272 19624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:38.467357 19625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:38.467327 19627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:17:38.468287 19619 server_base.cc:1034] running on GCE node
I20260924 16:17:38.469180 19619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:17:38.470697 19619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:17:38.471951 19619 hybrid_clock.cc:648] HybridClock initialized: now 1790266658471908 us; error 36 us; skew 500 ppm
I20260924 16:17:38.472316 19619 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:17:38 dist-test-slave-v5j9 krb5kdc[19605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266658, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:17:38.475468 19619 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 16:17:38.477051 19619 webserver.cc:460] Webserver started at http://127.0.0.1:46711/ using document root <none> and password file <none>
I20260924 16:17:38.477756 19619 fs_manager.cc:359] Metadata directory not provided
I20260924 16:17:38.477870 19619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:17:38.478124 19619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:17:38.480949 19619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790266323169505-30632-0/minicluster-data/master-0/data/instance:
uuid: "6d68d72deb3e43de8450fac38730ebb5"
format_stamp: "Formatted at 2026-09-24 16:17:38 on dist-test-slave-v5j9"
server_key: "24d81e656130ef853cbd898e45bff9a0"
server_key_iv: "18be15a475e4522ef5edc6573177c39f"
server_key_version: "encryptionkey@0"
I20260924 16:17:38.481732 19619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790266323169505-30632-0/minicluster-data/master-0/wal/instance:
uuid: "6d68d72deb3e43de8450fac38730ebb5"
format_stamp: "Formatted at 2026-09-24 16:17:38 on dist-test-slave-v5j9"
server_key: "24d81e656130ef853cbd898e45bff9a0"
server_key_iv: "18be15a475e4522ef5edc6573177c39f"
server_key_version: "encryptionkey@0"
I20260924 16:17:38.486654 19619 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260924 16:17:38.490041 19634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:17:38.491398 19619 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:17:38.491544 19619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790266323169505-30632-0/minicluster-data/master-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790266323169505-30632-0/minicluster-data/master-0/wal
uuid: "6d68d72deb3e43de8450fac38730ebb5"
format_stamp: "Formatted at 2026-09-24 16:17:38 on dist-test-slave-v5j9"
server_key: "24d81e656130ef853cbd898e45bff9a0"
server_key_iv: "18be15a475e4522ef5edc6573177c39f"
server_key_version: "encryptionkey@0"
I20260924 16:17:38.491704 19619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790266323169505-30632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790266323169505-30632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790266323169505-30632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:17:38.511988 19619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:17:38.516167 19619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:17:38.516410 19619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:17:38.536303 19619 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35613
I20260924 16:17:38.536331 19685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35613 every 8 connection(s)
I20260924 16:17:38.537899 19619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
I20260924 16:17:38.541925 19686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:17:38.547989 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 19619
I20260924 16:17:38.548192 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790266323169505-30632-0/minicluster-data/master-0/wal/instance
I20260924 16:17:38.548768 30632 external_mini_cluster.cc:1428] Setting key 0ef2344f4b1ac5af1697a3a46f95d38a
I20260924 16:17:38.553507 19686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d68d72deb3e43de8450fac38730ebb5: Bootstrap starting.
I20260924 16:17:38.557484 19686 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d68d72deb3e43de8450fac38730ebb5: Neither blocks nor log segments found. Creating new log.
I20260924 16:17:38.558902 19686 log.cc:824] T 00000000000000000000000000000000 P 6d68d72deb3e43de8450fac38730ebb5: Log is configured to *not* fsync() on all Append() calls
I20260924 16:17:38.562352 19686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d68d72deb3e43de8450fac38730ebb5: No bootstrap required, opened a new log
Sep 24 16:17:38 dist-test-slave-v5j9 krb5kdc[19605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266657, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 16:17:38.572505 19686 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d68d72deb3e43de8450fac38730ebb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d68d72deb3e43de8450fac38730ebb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35613 } }
I20260924 16:17:38.573024 19686 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d68d72deb3e43de8450fac38730ebb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:17:38.573148 19686 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d68d72deb3e43de8450fac38730ebb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d68d72deb3e43de8450fac38730ebb5, State: Initialized, Role: FOLLOWER
I20260924 16:17:38.573976 19686 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d68d72deb3e43de8450fac38730ebb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d68d72deb3e43de8450fac38730ebb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35613 } }
I20260924 16:17:38.574322 19686 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d68d72deb3e43de8450fac38730ebb5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:17:38.574481 19686 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d68d72deb3e43de8450fac38730ebb5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:17:38.574661 19686 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d68d72deb3e43de8450fac38730ebb5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:17:38.576066 19689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:38.551111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.241:55543 (local address 127.0.0.1:35613)
0924 16:17:38.552055 (+ 944us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:38.552074 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:38.552478 (+ 404us) server_negotiation.cc:407] Connection header received
0924 16:17:38.554927 (+ 2449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:38.554963 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:38.555542 (+ 579us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:38.556026 (+ 484us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:38.557387 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:38.559426 (+ 2039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:38.560988 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:38.562072 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:38.562265 (+ 193us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:38.566702 (+ 4437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:38.566714 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:38.566736 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:17:38.566802 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:38.571037 (+ 4235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:17:38.571869 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:17:38.571878 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:17:38.571887 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:17:38.572098 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:17:38.572531 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:17:38.572536 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:17:38.572539 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:17:38.573293 (+ 754us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:38.573657 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:38.574156 (+ 499us) server_negotiation.cc:299] Negotiation successful
0924 16:17:38.574540 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":671,"thread_start_us":391,"threads_started":1}
I20260924 16:17:38.577634 19686 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d68d72deb3e43de8450fac38730ebb5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d68d72deb3e43de8450fac38730ebb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35613 } }
I20260924 16:17:38.578126 19686 leader_election.cc:302] T 00000000000000000000000000000000 P 6d68d72deb3e43de8450fac38730ebb5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d68d72deb3e43de8450fac38730ebb5; no voters:
I20260924 16:17:38.579278 19686 leader_election.cc:288] T 00000000000000000000000000000000 P 6d68d72deb3e43de8450fac38730ebb5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:17:38.579401 19691 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d68d72deb3e43de8450fac38730ebb5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:17:38.580265 30632 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:17:38.581982 19686 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d68d72deb3e43de8450fac38730ebb5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:17:38.584558 19691 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d68d72deb3e43de8450fac38730ebb5 [term 1 LEADER]: Becoming Leader. State: Replica: 6d68d72deb3e43de8450fac38730ebb5, State: Running, Role: LEADER
I20260924 16:17:38.585394 19691 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d68d72deb3e43de8450fac38730ebb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d68d72deb3e43de8450fac38730ebb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35613 } }
I20260924 16:17:38.588934 19692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d68d72deb3e43de8450fac38730ebb5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d68d72deb3e43de8450fac38730ebb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d68d72deb3e43de8450fac38730ebb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35613 } } }
I20260924 16:17:38.589442 19692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d68d72deb3e43de8450fac38730ebb5 [sys.catalog]: This master's current role is: LEADER
I20260924 16:17:38.590049 19693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d68d72deb3e43de8450fac38730ebb5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d68d72deb3e43de8450fac38730ebb5. Latest consensus state: current_term: 1 leader_uuid: "6d68d72deb3e43de8450fac38730ebb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d68d72deb3e43de8450fac38730ebb5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35613 } } }
I20260924 16:17:38.590400 19693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d68d72deb3e43de8450fac38730ebb5 [sys.catalog]: This master's current role is: LEADER
I20260924 16:17:38.601037 19701 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:17:38.604564 19689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:38.585006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.241:38077 (local address 127.0.0.1:35613)
0924 16:17:38.585176 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:38.585183 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:38.586080 (+ 897us) server_negotiation.cc:407] Connection header received
0924 16:17:38.586283 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:38.586289 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:38.586389 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:38.586470 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:38.588346 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:38.589549 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:38.592023 (+ 2474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:38.592984 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:38.593109 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:38.597472 (+ 4363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:38.597477 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:38.597480 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:17:38.597514 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:38.600747 (+ 3233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:17:38.601733 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:17:38.601738 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:17:38.601742 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:17:38.601900 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:17:38.602523 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:17:38.602528 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:17:38.602531 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:17:38.602975 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:38.603259 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:38.603979 (+ 720us) server_negotiation.cc:299] Negotiation successful
0924 16:17:38.604170 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 16:17:38.607528 19701 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:17:38.617038 19701 catalog_manager.cc:1347] Generated new cluster ID: 3fe25f58e3004e31b64c50c3b06a1438
I20260924 16:17:38.617187 19701 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:17:38.634636 19701 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:17:38.636531 19701 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:17:38.646651 19701 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d68d72deb3e43de8450fac38730ebb5: Generated new TSK 0
I20260924 16:17:38.647794 19701 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:17:38.656252 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 19619
I20260924 16:17:38.657454 19681 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:17:38.749869 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 19619
W20260924 16:17:38.765142 19603 connection.cc:535] client connection to 127.0.0.1:35613 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35613): Connection reset by peer (error 104)
2026-09-24T16:17:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2062 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790266323169505-30632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:17:38 dist-test-slave-v5j9 krb5kdc[19720](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:17:38 dist-test-slave-v5j9 krb5kdc[19720](info): set up 2 sockets
Sep 24 16:17:38 dist-test-slave-v5j9 krb5kdc[19720](info): commencing operation
krb5kdc: starting...
W20260924 16:17:39.865319 30632 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Sep 24 16:17:40 dist-test-slave-v5j9 krb5kdc[19720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266660, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:17:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:17:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790266323169505-30632-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790266323169505-30632-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 16:17:40.128947 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790266323169505-30632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790266323169505-30632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790266323169505-30632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38571
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790266323169505-30632-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38571
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:17:40.462147 19734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:17:40.462491 19734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:17:40.462656 19734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:17:40.469882 19734 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:17:40.470048 19734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:17:40.470160 19734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:17:40.470274 19734 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:17:40.470352 19734 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:17:40.480458 19734 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790266323169505-30632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790266323169505-30632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38571
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790266323169505-30632-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790266323169505-30632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:17:40.481947 19734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:17:40.483407 19734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:17:40.495833 19739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:40.495888 19740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:40.495926 19743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:17:40.496691 19734 server_base.cc:1034] running on GCE node
I20260924 16:17:40.497601 19734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:17:40.499029 19734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:17:40.500319 19734 hybrid_clock.cc:648] HybridClock initialized: now 1790266660500256 us; error 42 us; skew 500 ppm
I20260924 16:17:40.500720 19734 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:17:40 dist-test-slave-v5j9 krb5kdc[19720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266660, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:17:40.512171 19734 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 16:17:40.513774 19734 webserver.cc:460] Webserver started at http://127.0.0.1:43027/ using document root <none> and password file <none>
I20260924 16:17:40.514444 19734 fs_manager.cc:359] Metadata directory not provided
I20260924 16:17:40.514555 19734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:17:40.514801 19734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:17:40.517822 19734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790266323169505-30632-0/minicluster-data/master-0/data/instance:
uuid: "8cda21f2d8d441fcbe14bc38a9f9bcef"
format_stamp: "Formatted at 2026-09-24 16:17:40 on dist-test-slave-v5j9"
server_key: "df077d8e557a90cf4b0d81b8fff3d6e0"
server_key_iv: "9433f0d9b503fb516befaa4f9105144a"
server_key_version: "encryptionkey@0"
I20260924 16:17:40.518538 19734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790266323169505-30632-0/minicluster-data/master-0/wal/instance:
uuid: "8cda21f2d8d441fcbe14bc38a9f9bcef"
format_stamp: "Formatted at 2026-09-24 16:17:40 on dist-test-slave-v5j9"
server_key: "df077d8e557a90cf4b0d81b8fff3d6e0"
server_key_iv: "9433f0d9b503fb516befaa4f9105144a"
server_key_version: "encryptionkey@0"
I20260924 16:17:40.523650 19734 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 16:17:40.527030 19750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:17:40.528362 19734 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 16:17:40.528538 19734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790266323169505-30632-0/minicluster-data/master-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790266323169505-30632-0/minicluster-data/master-0/wal
uuid: "8cda21f2d8d441fcbe14bc38a9f9bcef"
format_stamp: "Formatted at 2026-09-24 16:17:40 on dist-test-slave-v5j9"
server_key: "df077d8e557a90cf4b0d81b8fff3d6e0"
server_key_iv: "9433f0d9b503fb516befaa4f9105144a"
server_key_version: "encryptionkey@0"
I20260924 16:17:40.528705 19734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790266323169505-30632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790266323169505-30632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790266323169505-30632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:17:40.544363 19734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:17:40.548720 19734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:17:40.549019 19734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:17:40.568094 19734 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38571
I20260924 16:17:40.568130 19801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38571 every 8 connection(s)
I20260924 16:17:40.569662 19734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
I20260924 16:17:40.573781 19802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:17:40.573992 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 19734
I20260924 16:17:40.574188 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790266323169505-30632-0/minicluster-data/master-0/wal/instance
I20260924 16:17:40.574640 30632 external_mini_cluster.cc:1428] Setting key f52d57a47f50bae56127ab92d5d9fcca
I20260924 16:17:40.589542 19802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cda21f2d8d441fcbe14bc38a9f9bcef: Bootstrap starting.
I20260924 16:17:40.589672 19806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:40.576581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.241:57579 (local address 127.0.0.1:38571)
0924 16:17:40.577522 (+ 941us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:40.577538 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:40.577599 (+ 61us) server_negotiation.cc:407] Connection header received
0924 16:17:40.579329 (+ 1730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:40.579365 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:40.579930 (+ 565us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:40.580413 (+ 483us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:40.580970 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:40.583017 (+ 2047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:40.584331 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:40.585406 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:40.585618 (+ 212us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:40.586786 (+ 1168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:40.586796 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:40.586820 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:17:40.587579 (+ 759us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:40.587675 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:17:40.587705 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:40.587855 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:40.588107 (+ 252us) server_negotiation.cc:299] Negotiation successful
0924 16:17:40.588302 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":373,"threads_started":1}
I20260924 16:17:40.592916 19802 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8cda21f2d8d441fcbe14bc38a9f9bcef: Neither blocks nor log segments found. Creating new log.
I20260924 16:17:40.593535 30632 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:17:40.594127 19802 log.cc:824] T 00000000000000000000000000000000 P 8cda21f2d8d441fcbe14bc38a9f9bcef: Log is configured to *not* fsync() on all Append() calls
I20260924 16:17:40.597173 19802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cda21f2d8d441fcbe14bc38a9f9bcef: No bootstrap required, opened a new log
I20260924 16:17:40.605399 19806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:40.598511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.241:35285 (local address 127.0.0.1:38571)
0924 16:17:40.598694 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:40.598699 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:40.599040 (+ 341us) server_negotiation.cc:407] Connection header received
0924 16:17:40.599214 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:40.599225 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:40.599305 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:40.599420 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:40.599942 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:40.601128 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:40.602589 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:40.603304 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:40.603439 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:40.604555 (+ 1116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:40.604562 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:40.604567 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:17:40.604593 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:40.604639 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:17:40.604659 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:40.604779 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:40.604949 (+ 170us) server_negotiation.cc:299] Negotiation successful
0924 16:17:40.605008 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 16:17:40.606951 19802 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8cda21f2d8d441fcbe14bc38a9f9bcef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cda21f2d8d441fcbe14bc38a9f9bcef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38571 } }
I20260924 16:17:40.607394 19802 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8cda21f2d8d441fcbe14bc38a9f9bcef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:17:40.607476 19802 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8cda21f2d8d441fcbe14bc38a9f9bcef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cda21f2d8d441fcbe14bc38a9f9bcef, State: Initialized, Role: FOLLOWER
I20260924 16:17:40.608002 19802 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8cda21f2d8d441fcbe14bc38a9f9bcef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cda21f2d8d441fcbe14bc38a9f9bcef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38571 } }
I20260924 16:17:40.608206 19802 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8cda21f2d8d441fcbe14bc38a9f9bcef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:17:40.608309 19802 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8cda21f2d8d441fcbe14bc38a9f9bcef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:17:40.608431 19802 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8cda21f2d8d441fcbe14bc38a9f9bcef [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:17:40.610723 19802 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8cda21f2d8d441fcbe14bc38a9f9bcef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cda21f2d8d441fcbe14bc38a9f9bcef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38571 } }
I20260924 16:17:40.611179 19802 leader_election.cc:302] T 00000000000000000000000000000000 P 8cda21f2d8d441fcbe14bc38a9f9bcef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8cda21f2d8d441fcbe14bc38a9f9bcef; no voters:
I20260924 16:17:40.611845 19802 leader_election.cc:288] T 00000000000000000000000000000000 P 8cda21f2d8d441fcbe14bc38a9f9bcef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:17:40.611992 19813 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8cda21f2d8d441fcbe14bc38a9f9bcef [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:17:40.612911 19813 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8cda21f2d8d441fcbe14bc38a9f9bcef [term 1 LEADER]: Becoming Leader. State: Replica: 8cda21f2d8d441fcbe14bc38a9f9bcef, State: Running, Role: LEADER
I20260924 16:17:40.613543 19813 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8cda21f2d8d441fcbe14bc38a9f9bcef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cda21f2d8d441fcbe14bc38a9f9bcef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38571 } }
I20260924 16:17:40.614311 19802 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8cda21f2d8d441fcbe14bc38a9f9bcef [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:17:40.616122 19814 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cda21f2d8d441fcbe14bc38a9f9bcef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8cda21f2d8d441fcbe14bc38a9f9bcef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cda21f2d8d441fcbe14bc38a9f9bcef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38571 } } }
I20260924 16:17:40.616453 19814 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cda21f2d8d441fcbe14bc38a9f9bcef [sys.catalog]: This master's current role is: LEADER
I20260924 16:17:40.617297 19819 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:17:40.617139 19815 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cda21f2d8d441fcbe14bc38a9f9bcef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8cda21f2d8d441fcbe14bc38a9f9bcef. Latest consensus state: current_term: 1 leader_uuid: "8cda21f2d8d441fcbe14bc38a9f9bcef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cda21f2d8d441fcbe14bc38a9f9bcef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38571 } } }
I20260924 16:17:40.618340 19815 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cda21f2d8d441fcbe14bc38a9f9bcef [sys.catalog]: This master's current role is: LEADER
I20260924 16:17:40.622864 19819 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:17:40.633628 19819 catalog_manager.cc:1347] Generated new cluster ID: e9762c7cd1fe4caf9523aa2d9b858af7
I20260924 16:17:40.633735 19819 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:17:40.649488 19819 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:17:40.650841 19819 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:17:40.657940 19819 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8cda21f2d8d441fcbe14bc38a9f9bcef: Generated new TSK 0
I20260924 16:17:40.658785 19819 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:17:40.666064 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 19734
W20260924 16:17:40.667292 19797 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.241:57579
W20260924 16:17:40.668044 30632 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 19734: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 16:17:40.668203 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 19734
W20260924 16:17:40.686203 19718 connection.cc:535] client connection to 127.0.0.1:38571 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38571): Connection reset by peer (error 104)
2026-09-24T16:17:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1921 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790266323169505-30632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:17:40 dist-test-slave-v5j9 krb5kdc[19835](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:17:40 dist-test-slave-v5j9 krb5kdc[19835](info): set up 2 sockets
Sep 24 16:17:40 dist-test-slave-v5j9 krb5kdc[19835](info): commencing operation
krb5kdc: starting...
W20260924 16:17:41.785466 30632 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Sep 24 16:17:41 dist-test-slave-v5j9 krb5kdc[19835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266661, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:17:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:17:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790266323169505-30632-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790266323169505-30632-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 16:17:42.060683 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790266323169505-30632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790266323169505-30632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790266323169505-30632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33037
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790266323169505-30632-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33037
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:17:42.422153 19849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:17:42.422482 19849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:17:42.422662 19849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:17:42.430364 19849 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:17:42.430517 19849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:17:42.430619 19849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:17:42.430724 19849 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:17:42.430810 19849 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:17:42.441063 19849 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790266323169505-30632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790266323169505-30632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33037
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790266323169505-30632-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790266323169505-30632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:17:42.442492 19849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:17:42.443902 19849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:17:42.456267 19857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:42.456374 19855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:42.456338 19854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:17:42.457541 19849 server_base.cc:1034] running on GCE node
I20260924 16:17:42.458348 19849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:17:42.459720 19849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:17:42.460953 19849 hybrid_clock.cc:648] HybridClock initialized: now 1790266662460917 us; error 57 us; skew 500 ppm
I20260924 16:17:42.461308 19849 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:17:42 dist-test-slave-v5j9 krb5kdc[19835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266662, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:17:42.464399 19849 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 16:17:42.465973 19849 webserver.cc:460] Webserver started at http://127.0.0.1:33059/ using document root <none> and password file <none>
I20260924 16:17:42.466683 19849 fs_manager.cc:359] Metadata directory not provided
I20260924 16:17:42.466833 19849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:17:42.467159 19849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:17:42.470256 19849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790266323169505-30632-0/minicluster-data/master-0/data/instance:
uuid: "3ef5ce2a15b24f9e8a60eed2d15b5d5c"
format_stamp: "Formatted at 2026-09-24 16:17:42 on dist-test-slave-v5j9"
server_key: "aca5623d1a5de58da641a509bec142cc"
server_key_iv: "5a9a1625d0d9d905e1fcda34f26357e7"
server_key_version: "encryptionkey@0"
I20260924 16:17:42.471050 19849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790266323169505-30632-0/minicluster-data/master-0/wal/instance:
uuid: "3ef5ce2a15b24f9e8a60eed2d15b5d5c"
format_stamp: "Formatted at 2026-09-24 16:17:42 on dist-test-slave-v5j9"
server_key: "aca5623d1a5de58da641a509bec142cc"
server_key_iv: "5a9a1625d0d9d905e1fcda34f26357e7"
server_key_version: "encryptionkey@0"
I20260924 16:17:42.476600 19849 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 16:17:42.480212 19864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:17:42.481590 19849 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 16:17:42.481778 19849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790266323169505-30632-0/minicluster-data/master-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790266323169505-30632-0/minicluster-data/master-0/wal
uuid: "3ef5ce2a15b24f9e8a60eed2d15b5d5c"
format_stamp: "Formatted at 2026-09-24 16:17:42 on dist-test-slave-v5j9"
server_key: "aca5623d1a5de58da641a509bec142cc"
server_key_iv: "5a9a1625d0d9d905e1fcda34f26357e7"
server_key_version: "encryptionkey@0"
I20260924 16:17:42.481983 19849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790266323169505-30632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790266323169505-30632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790266323169505-30632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:17:42.487814 19849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:17:42.491871 19849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:17:42.492187 19849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:17:42.511515 19849 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33037
I20260924 16:17:42.511561 19918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33037 every 8 connection(s)
I20260924 16:17:42.513087 19849 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
I20260924 16:17:42.516285 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 19849
I20260924 16:17:42.516464 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790266323169505-30632-0/minicluster-data/master-0/wal/instance
I20260924 16:17:42.517011 30632 external_mini_cluster.cc:1428] Setting key 868f48173077cfa78c6b8f2394eb68e6
I20260924 16:17:42.517653 19919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:17:42.524956 19922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:42.519015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.241:39271 (local address 127.0.0.1:33037)
0924 16:17:42.519785 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:42.519798 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:42.519837 (+ 39us) server_negotiation.cc:407] Connection header received
0924 16:17:42.521424 (+ 1587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:42.521447 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:42.521813 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:42.522238 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:42.523293 (+ 1055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:42.523299 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:42.523312 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:17:42.523326 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:42.523400 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:17:42.523420 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:42.523670 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:42.523827 (+ 157us) server_negotiation.cc:299] Negotiation successful
0924 16:17:42.524024 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":332,"threads_started":1}
I20260924 16:17:42.528616 30632 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:17:42.530714 19919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ef5ce2a15b24f9e8a60eed2d15b5d5c: Bootstrap starting.
I20260924 16:17:42.534300 19919 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ef5ce2a15b24f9e8a60eed2d15b5d5c: Neither blocks nor log segments found. Creating new log.
I20260924 16:17:42.535521 19919 log.cc:824] T 00000000000000000000000000000000 P 3ef5ce2a15b24f9e8a60eed2d15b5d5c: Log is configured to *not* fsync() on all Append() calls
I20260924 16:17:42.535883 19922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:42.533193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.241:41029 (local address 127.0.0.1:33037)
0924 16:17:42.533408 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:42.533417 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:42.533842 (+ 425us) server_negotiation.cc:407] Connection header received
0924 16:17:42.534044 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:42.534049 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:42.534110 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:42.534224 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:42.535192 (+ 968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:42.535197 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:42.535202 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:17:42.535220 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:42.535272 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:17:42.535288 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:42.535410 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:42.535540 (+ 130us) server_negotiation.cc:299] Negotiation successful
0924 16:17:42.535605 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 16:17:42.538772 19919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ef5ce2a15b24f9e8a60eed2d15b5d5c: No bootstrap required, opened a new log
I20260924 16:17:42.547144 19919 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ef5ce2a15b24f9e8a60eed2d15b5d5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ef5ce2a15b24f9e8a60eed2d15b5d5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33037 } }
I20260924 16:17:42.547678 19919 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ef5ce2a15b24f9e8a60eed2d15b5d5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:17:42.547808 19919 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ef5ce2a15b24f9e8a60eed2d15b5d5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ef5ce2a15b24f9e8a60eed2d15b5d5c, State: Initialized, Role: FOLLOWER
I20260924 16:17:42.548532 19919 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ef5ce2a15b24f9e8a60eed2d15b5d5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ef5ce2a15b24f9e8a60eed2d15b5d5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33037 } }
I20260924 16:17:42.548872 19919 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ef5ce2a15b24f9e8a60eed2d15b5d5c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:17:42.549026 19919 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ef5ce2a15b24f9e8a60eed2d15b5d5c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:17:42.549214 19919 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ef5ce2a15b24f9e8a60eed2d15b5d5c [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:17:42.551424 19919 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ef5ce2a15b24f9e8a60eed2d15b5d5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ef5ce2a15b24f9e8a60eed2d15b5d5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33037 } }
I20260924 16:17:42.551940 19919 leader_election.cc:302] T 00000000000000000000000000000000 P 3ef5ce2a15b24f9e8a60eed2d15b5d5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ef5ce2a15b24f9e8a60eed2d15b5d5c; no voters:
I20260924 16:17:42.552527 19919 leader_election.cc:288] T 00000000000000000000000000000000 P 3ef5ce2a15b24f9e8a60eed2d15b5d5c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:17:42.552677 19929 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ef5ce2a15b24f9e8a60eed2d15b5d5c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:17:42.553407 19929 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ef5ce2a15b24f9e8a60eed2d15b5d5c [term 1 LEADER]: Becoming Leader. State: Replica: 3ef5ce2a15b24f9e8a60eed2d15b5d5c, State: Running, Role: LEADER
I20260924 16:17:42.554001 19929 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ef5ce2a15b24f9e8a60eed2d15b5d5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ef5ce2a15b24f9e8a60eed2d15b5d5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33037 } }
I20260924 16:17:42.554700 19919 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ef5ce2a15b24f9e8a60eed2d15b5d5c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:17:42.556998 19931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ef5ce2a15b24f9e8a60eed2d15b5d5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ef5ce2a15b24f9e8a60eed2d15b5d5c. Latest consensus state: current_term: 1 leader_uuid: "3ef5ce2a15b24f9e8a60eed2d15b5d5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ef5ce2a15b24f9e8a60eed2d15b5d5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33037 } } }
I20260924 16:17:42.556970 19930 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ef5ce2a15b24f9e8a60eed2d15b5d5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ef5ce2a15b24f9e8a60eed2d15b5d5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ef5ce2a15b24f9e8a60eed2d15b5d5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33037 } } }
I20260924 16:17:42.557453 19931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ef5ce2a15b24f9e8a60eed2d15b5d5c [sys.catalog]: This master's current role is: LEADER
I20260924 16:17:42.557504 19930 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ef5ce2a15b24f9e8a60eed2d15b5d5c [sys.catalog]: This master's current role is: LEADER
I20260924 16:17:42.558504 19934 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:17:42.571403 19934 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:17:42.581511 19934 catalog_manager.cc:1347] Generated new cluster ID: 41b3b4ae74344659b40253e97841a70a
I20260924 16:17:42.581678 19934 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:17:42.608487 19934 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:17:42.608927 19934 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:17:42.620101 19934 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ef5ce2a15b24f9e8a60eed2d15b5d5c: Generated new TSK 0
I20260924 16:17:42.620965 19934 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:17:42.633488 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 19849
W20260924 16:17:42.634625 19914 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.241:39271
W20260924 16:17:42.635424 30632 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 19849: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 16:17:42.635569 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 19849
2026-09-24T16:17:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1966 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790266323169505-30632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:17:42 dist-test-slave-v5j9 krb5kdc[19950](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:17:42 dist-test-slave-v5j9 krb5kdc[19950](info): set up 2 sockets
Sep 24 16:17:42 dist-test-slave-v5j9 krb5kdc[19950](info): commencing operation
krb5kdc: starting...
W20260924 16:17:43.745692 30632 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Sep 24 16:17:43 dist-test-slave-v5j9 krb5kdc[19950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:17:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:17:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790266323169505-30632-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790266323169505-30632-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 16:17:44.017023 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790266323169505-30632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790266323169505-30632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790266323169505-30632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40695
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790266323169505-30632-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40695
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:17:44.353322 19963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:17:44.353657 19963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:17:44.353826 19963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:17:44.361124 19963 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:17:44.361248 19963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:17:44.361294 19963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:17:44.361406 19963 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:17:44.361476 19963 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:17:44.371784 19963 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790266323169505-30632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790266323169505-30632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40695
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790266323169505-30632-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790266323169505-30632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:17:44.373164 19963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:17:44.374571 19963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:17:44.387791 19968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:44.387840 19969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:44.388366 19972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:17:44.388552 19963 server_base.cc:1034] running on GCE node
I20260924 16:17:44.389768 19963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:17:44.391374 19963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:17:44.392616 19963 hybrid_clock.cc:648] HybridClock initialized: now 1790266664392576 us; error 58 us; skew 500 ppm
I20260924 16:17:44.393018 19963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:17:44 dist-test-slave-v5j9 krb5kdc[19950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266664, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:17:44.396310 19963 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 16:17:44.397907 19963 webserver.cc:460] Webserver started at http://127.0.0.1:33667/ using document root <none> and password file <none>
I20260924 16:17:44.398644 19963 fs_manager.cc:359] Metadata directory not provided
I20260924 16:17:44.398804 19963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:17:44.399125 19963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:17:44.402333 19963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790266323169505-30632-0/minicluster-data/master-0/data/instance:
uuid: "cf1f9f6f2c8041fd9cd5bba29720a654"
format_stamp: "Formatted at 2026-09-24 16:17:44 on dist-test-slave-v5j9"
server_key: "b6a139d16a8953c69082d20c7c076cd7"
server_key_iv: "313f96cc8865392483ea8e994b4b0d9e"
server_key_version: "encryptionkey@0"
I20260924 16:17:44.403169 19963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790266323169505-30632-0/minicluster-data/master-0/wal/instance:
uuid: "cf1f9f6f2c8041fd9cd5bba29720a654"
format_stamp: "Formatted at 2026-09-24 16:17:44 on dist-test-slave-v5j9"
server_key: "b6a139d16a8953c69082d20c7c076cd7"
server_key_iv: "313f96cc8865392483ea8e994b4b0d9e"
server_key_version: "encryptionkey@0"
I20260924 16:17:44.408478 19963 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260924 16:17:44.411892 19980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:17:44.413179 19963 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 16:17:44.413400 19963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790266323169505-30632-0/minicluster-data/master-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790266323169505-30632-0/minicluster-data/master-0/wal
uuid: "cf1f9f6f2c8041fd9cd5bba29720a654"
format_stamp: "Formatted at 2026-09-24 16:17:44 on dist-test-slave-v5j9"
server_key: "b6a139d16a8953c69082d20c7c076cd7"
server_key_iv: "313f96cc8865392483ea8e994b4b0d9e"
server_key_version: "encryptionkey@0"
I20260924 16:17:44.413595 19963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790266323169505-30632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790266323169505-30632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790266323169505-30632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:17:44.430202 19963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:17:44.434762 19963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:17:44.435077 19963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:17:44.453516 20031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40695 every 8 connection(s)
I20260924 16:17:44.453516 19963 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40695
I20260924 16:17:44.455137 19963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
I20260924 16:17:44.459195 20032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:17:44.464008 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 19963
I20260924 16:17:44.464159 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790266323169505-30632-0/minicluster-data/master-0/wal/instance
I20260924 16:17:44.464622 30632 external_mini_cluster.cc:1428] Setting key 9c8b13fb40a379ecbaa8f826562d46fd
I20260924 16:17:44.471042 20032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf1f9f6f2c8041fd9cd5bba29720a654: Bootstrap starting.
I20260924 16:17:44.474900 20032 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf1f9f6f2c8041fd9cd5bba29720a654: Neither blocks nor log segments found. Creating new log.
I20260924 16:17:44.476141 20032 log.cc:824] T 00000000000000000000000000000000 P cf1f9f6f2c8041fd9cd5bba29720a654: Log is configured to *not* fsync() on all Append() calls
I20260924 16:17:44.479743 20032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf1f9f6f2c8041fd9cd5bba29720a654: No bootstrap required, opened a new log
Sep 24 16:17:44 dist-test-slave-v5j9 krb5kdc[19950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 16:17:44.487859 20032 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf1f9f6f2c8041fd9cd5bba29720a654 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf1f9f6f2c8041fd9cd5bba29720a654" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40695 } }
I20260924 16:17:44.488289 20032 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf1f9f6f2c8041fd9cd5bba29720a654 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:17:44.488396 20032 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf1f9f6f2c8041fd9cd5bba29720a654 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf1f9f6f2c8041fd9cd5bba29720a654, State: Initialized, Role: FOLLOWER
I20260924 16:17:44.489065 20035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:44.466490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.241:58027 (local address 127.0.0.1:40695)
0924 16:17:44.467288 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:44.467302 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:44.467345 (+ 43us) server_negotiation.cc:407] Connection header received
0924 16:17:44.469182 (+ 1837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:44.469207 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:44.469719 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:44.470058 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:44.472002 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:44.473493 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:44.475310 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:44.475976 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:44.476105 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:44.481098 (+ 4993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:44.481106 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:44.481122 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:17:44.481160 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:44.484439 (+ 3279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:17:44.485259 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:17:44.485265 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:17:44.485272 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:17:44.485524 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:17:44.486010 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:17:44.486025 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:17:44.486027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:17:44.486592 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:44.486954 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:44.487462 (+ 508us) server_negotiation.cc:299] Negotiation successful
0924 16:17:44.487783 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":353,"threads_started":1}
I20260924 16:17:44.489045 20032 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf1f9f6f2c8041fd9cd5bba29720a654 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf1f9f6f2c8041fd9cd5bba29720a654" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40695 } }
I20260924 16:17:44.489540 20032 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf1f9f6f2c8041fd9cd5bba29720a654 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:17:44.489702 20032 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf1f9f6f2c8041fd9cd5bba29720a654 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:17:44.489905 20032 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf1f9f6f2c8041fd9cd5bba29720a654 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:17:44.492796 20032 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf1f9f6f2c8041fd9cd5bba29720a654 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf1f9f6f2c8041fd9cd5bba29720a654" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40695 } }
I20260924 16:17:44.493494 20032 leader_election.cc:302] T 00000000000000000000000000000000 P cf1f9f6f2c8041fd9cd5bba29720a654 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf1f9f6f2c8041fd9cd5bba29720a654; no voters:
I20260924 16:17:44.493574 30632 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:17:44.494156 20032 leader_election.cc:288] T 00000000000000000000000000000000 P cf1f9f6f2c8041fd9cd5bba29720a654 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:17:44.494266 20037 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf1f9f6f2c8041fd9cd5bba29720a654 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:17:44.496299 20032 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf1f9f6f2c8041fd9cd5bba29720a654 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:17:44.502816 20037 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf1f9f6f2c8041fd9cd5bba29720a654 [term 1 LEADER]: Becoming Leader. State: Replica: cf1f9f6f2c8041fd9cd5bba29720a654, State: Running, Role: LEADER
I20260924 16:17:44.504218 20037 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf1f9f6f2c8041fd9cd5bba29720a654 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf1f9f6f2c8041fd9cd5bba29720a654" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40695 } }
I20260924 16:17:44.507426 20039 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf1f9f6f2c8041fd9cd5bba29720a654 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf1f9f6f2c8041fd9cd5bba29720a654. Latest consensus state: current_term: 1 leader_uuid: "cf1f9f6f2c8041fd9cd5bba29720a654" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf1f9f6f2c8041fd9cd5bba29720a654" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40695 } } }
I20260924 16:17:44.507820 20039 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf1f9f6f2c8041fd9cd5bba29720a654 [sys.catalog]: This master's current role is: LEADER
I20260924 16:17:44.508325 20038 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf1f9f6f2c8041fd9cd5bba29720a654 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf1f9f6f2c8041fd9cd5bba29720a654" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf1f9f6f2c8041fd9cd5bba29720a654" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40695 } } }
I20260924 16:17:44.508584 20038 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf1f9f6f2c8041fd9cd5bba29720a654 [sys.catalog]: This master's current role is: LEADER
I20260924 16:17:44.510866 20054 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:17:44.516531 20054 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:17:44.522807 20035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:44.503204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.241:38579 (local address 127.0.0.1:40695)
0924 16:17:44.503416 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:44.503424 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:44.504335 (+ 911us) server_negotiation.cc:407] Connection header received
0924 16:17:44.504623 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:44.504647 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:44.504732 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:44.504844 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:44.507228 (+ 2384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:44.508482 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:44.510645 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:44.511550 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:44.511693 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:44.515050 (+ 3357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:44.515057 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:44.515063 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:17:44.515105 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:44.519052 (+ 3947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:17:44.520025 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:17:44.520031 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:17:44.520035 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:17:44.520186 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:17:44.520813 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:17:44.520819 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:17:44.520823 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:17:44.521229 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:44.521496 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:44.522214 (+ 718us) server_negotiation.cc:299] Negotiation successful
0924 16:17:44.522435 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260924 16:17:44.531949 20054 catalog_manager.cc:1347] Generated new cluster ID: 5de378ce989e47d59044cd3318ab350b
I20260924 16:17:44.532071 20054 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:17:44.542379 20054 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:17:44.544327 20054 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:17:44.555770 20054 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf1f9f6f2c8041fd9cd5bba29720a654: Generated new TSK 0
I20260924 16:17:44.557027 20054 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:17:44.568660 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 19963
I20260924 16:17:44.569944 20027 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:17:44.661923 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 19963
W20260924 16:17:44.680572 19948 connection.cc:535] client connection to 127.0.0.1:40695 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40695): Connection reset by peer (error 104)
2026-09-24T16:17:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2027 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790266323169505-30632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:17:44 dist-test-slave-v5j9 krb5kdc[20065](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:17:44 dist-test-slave-v5j9 krb5kdc[20065](info): set up 2 sockets
Sep 24 16:17:44 dist-test-slave-v5j9 krb5kdc[20065](info): commencing operation
krb5kdc: starting...
W20260924 16:17:45.785069 30632 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Sep 24 16:17:45 dist-test-slave-v5j9 krb5kdc[20065](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266665, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:17:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:17:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790266323169505-30632-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790266323169505-30632-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 16:17:46.060500 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790266323169505-30632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790266323169505-30632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790266323169505-30632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36123
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790266323169505-30632-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36123
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:17:46.397938 20078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:17:46.398240 20078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:17:46.398344 20078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:17:46.405826 20078 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:17:46.405942 20078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:17:46.405993 20078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:17:46.406054 20078 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:17:46.406111 20078 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:17:46.415884 20078 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790266323169505-30632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790266323169505-30632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36123
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790266323169505-30632-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790266323169505-30632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:17:46.417177 20078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:17:46.418511 20078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:17:46.430878 20084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:46.430800 20086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:46.430830 20083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:17:46.431263 20078 server_base.cc:1034] running on GCE node
I20260924 16:17:46.432180 20078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:17:46.433652 20078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:17:46.434896 20078 hybrid_clock.cc:648] HybridClock initialized: now 1790266666434833 us; error 49 us; skew 500 ppm
I20260924 16:17:46.435277 20078 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:17:46 dist-test-slave-v5j9 krb5kdc[20065](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266666, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:17:46.447197 20078 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 16:17:46.448747 20078 webserver.cc:460] Webserver started at http://127.0.0.1:36523/ using document root <none> and password file <none>
I20260924 16:17:46.449527 20078 fs_manager.cc:359] Metadata directory not provided
I20260924 16:17:46.449637 20078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:17:46.449894 20078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:17:46.452929 20078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790266323169505-30632-0/minicluster-data/master-0/data/instance:
uuid: "9bff9d65c5d6442b946d58c8ca75c1bb"
format_stamp: "Formatted at 2026-09-24 16:17:46 on dist-test-slave-v5j9"
server_key: "c073c7d2a5d82885c961e1ea7cf63a81"
server_key_iv: "571ded6fff719e8aa0ec8b1a42837f78"
server_key_version: "encryptionkey@0"
I20260924 16:17:46.453754 20078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790266323169505-30632-0/minicluster-data/master-0/wal/instance:
uuid: "9bff9d65c5d6442b946d58c8ca75c1bb"
format_stamp: "Formatted at 2026-09-24 16:17:46 on dist-test-slave-v5j9"
server_key: "c073c7d2a5d82885c961e1ea7cf63a81"
server_key_iv: "571ded6fff719e8aa0ec8b1a42837f78"
server_key_version: "encryptionkey@0"
I20260924 16:17:46.458942 20078 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 16:17:46.462234 20093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:17:46.463563 20078 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 16:17:46.463745 20078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790266323169505-30632-0/minicluster-data/master-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790266323169505-30632-0/minicluster-data/master-0/wal
uuid: "9bff9d65c5d6442b946d58c8ca75c1bb"
format_stamp: "Formatted at 2026-09-24 16:17:46 on dist-test-slave-v5j9"
server_key: "c073c7d2a5d82885c961e1ea7cf63a81"
server_key_iv: "571ded6fff719e8aa0ec8b1a42837f78"
server_key_version: "encryptionkey@0"
I20260924 16:17:46.463932 20078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790266323169505-30632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790266323169505-30632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790266323169505-30632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:17:46.488423 20078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:17:46.492596 20078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:17:46.492889 20078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:17:46.511861 20078 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36123
I20260924 16:17:46.511893 20147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36123 every 8 connection(s)
I20260924 16:17:46.513371 20078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
I20260924 16:17:46.516295 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 20078
I20260924 16:17:46.516480 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790266323169505-30632-0/minicluster-data/master-0/wal/instance
I20260924 16:17:46.517064 30632 external_mini_cluster.cc:1428] Setting key ea59edf88ff202afe34bcbc056dc10ab
I20260924 16:17:46.517928 20148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:17:46.530747 20151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:46.518970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.241:57301 (local address 127.0.0.1:36123)
0924 16:17:46.519771 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:46.519784 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:46.519827 (+ 43us) server_negotiation.cc:407] Connection header received
0924 16:17:46.521478 (+ 1651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:46.521504 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:46.521942 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:46.522292 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:46.522916 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:46.524300 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:46.526238 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:46.526901 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:46.527036 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:46.528380 (+ 1344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:46.528387 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:46.528401 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:17:46.528963 (+ 562us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:46.529037 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:17:46.529064 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:46.529199 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:46.529562 (+ 363us) server_negotiation.cc:299] Negotiation successful
0924 16:17:46.529761 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":351,"threads_started":1}
I20260924 16:17:46.531876 20148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bff9d65c5d6442b946d58c8ca75c1bb: Bootstrap starting.
I20260924 16:17:46.534621 30632 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:17:46.535172 20148 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9bff9d65c5d6442b946d58c8ca75c1bb: Neither blocks nor log segments found. Creating new log.
I20260924 16:17:46.536367 20148 log.cc:824] T 00000000000000000000000000000000 P 9bff9d65c5d6442b946d58c8ca75c1bb: Log is configured to *not* fsync() on all Append() calls
I20260924 16:17:46.539677 20148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bff9d65c5d6442b946d58c8ca75c1bb: No bootstrap required, opened a new log
I20260924 16:17:46.546514 20151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:46.539753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.241:37379 (local address 127.0.0.1:36123)
0924 16:17:46.539919 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:46.539923 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:46.540339 (+ 416us) server_negotiation.cc:407] Connection header received
0924 16:17:46.540524 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:46.540527 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:46.540589 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:46.540666 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:46.541265 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:46.542093 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:46.543959 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:46.544409 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:46.544501 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:46.545717 (+ 1216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:46.545721 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:46.545724 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:17:46.545740 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:46.545779 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:17:46.545794 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:46.545881 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:46.546098 (+ 217us) server_negotiation.cc:299] Negotiation successful
0924 16:17:46.546180 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 16:17:46.549072 20148 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9bff9d65c5d6442b946d58c8ca75c1bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bff9d65c5d6442b946d58c8ca75c1bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36123 } }
I20260924 16:17:46.549544 20148 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9bff9d65c5d6442b946d58c8ca75c1bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:17:46.549665 20148 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9bff9d65c5d6442b946d58c8ca75c1bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bff9d65c5d6442b946d58c8ca75c1bb, State: Initialized, Role: FOLLOWER
I20260924 16:17:46.550315 20148 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9bff9d65c5d6442b946d58c8ca75c1bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bff9d65c5d6442b946d58c8ca75c1bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36123 } }
I20260924 16:17:46.550559 20148 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9bff9d65c5d6442b946d58c8ca75c1bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:17:46.550705 20148 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9bff9d65c5d6442b946d58c8ca75c1bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:17:46.550899 20148 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9bff9d65c5d6442b946d58c8ca75c1bb [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:17:46.553086 20148 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9bff9d65c5d6442b946d58c8ca75c1bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bff9d65c5d6442b946d58c8ca75c1bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36123 } }
I20260924 16:17:46.553663 20148 leader_election.cc:302] T 00000000000000000000000000000000 P 9bff9d65c5d6442b946d58c8ca75c1bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9bff9d65c5d6442b946d58c8ca75c1bb; no voters:
I20260924 16:17:46.554309 20148 leader_election.cc:288] T 00000000000000000000000000000000 P 9bff9d65c5d6442b946d58c8ca75c1bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:17:46.554441 20158 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9bff9d65c5d6442b946d58c8ca75c1bb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:17:46.555260 20158 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9bff9d65c5d6442b946d58c8ca75c1bb [term 1 LEADER]: Becoming Leader. State: Replica: 9bff9d65c5d6442b946d58c8ca75c1bb, State: Running, Role: LEADER
I20260924 16:17:46.555811 20158 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9bff9d65c5d6442b946d58c8ca75c1bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bff9d65c5d6442b946d58c8ca75c1bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36123 } }
I20260924 16:17:46.556536 20148 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9bff9d65c5d6442b946d58c8ca75c1bb [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:17:46.558634 20160 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bff9d65c5d6442b946d58c8ca75c1bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9bff9d65c5d6442b946d58c8ca75c1bb. Latest consensus state: current_term: 1 leader_uuid: "9bff9d65c5d6442b946d58c8ca75c1bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bff9d65c5d6442b946d58c8ca75c1bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36123 } } }
I20260924 16:17:46.559095 20160 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bff9d65c5d6442b946d58c8ca75c1bb [sys.catalog]: This master's current role is: LEADER
I20260924 16:17:46.559551 20159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bff9d65c5d6442b946d58c8ca75c1bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9bff9d65c5d6442b946d58c8ca75c1bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bff9d65c5d6442b946d58c8ca75c1bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36123 } } }
I20260924 16:17:46.559842 20159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bff9d65c5d6442b946d58c8ca75c1bb [sys.catalog]: This master's current role is: LEADER
I20260924 16:17:46.566272 20165 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:17:46.572870 20165 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:17:46.583078 20165 catalog_manager.cc:1347] Generated new cluster ID: 0100f1ba7037493d9fb4d3cd8cd683e8
I20260924 16:17:46.583201 20165 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:17:46.605150 20165 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:17:46.607028 20165 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:17:46.614707 20165 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9bff9d65c5d6442b946d58c8ca75c1bb: Generated new TSK 0
I20260924 16:17:46.615638 20165 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:17:46.633867 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 20078
W20260924 16:17:46.635326 20143 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.241:57301
W20260924 16:17:46.636309 30632 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 20078: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 16:17:46.636493 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 20078
W20260924 16:17:46.655582 20063 connection.cc:535] client connection to 127.0.0.1:36123 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36123): Connection reset by peer (error 104)
2026-09-24T16:17:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1974 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790266323169505-30632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:17:46 dist-test-slave-v5j9 krb5kdc[20180](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:17:46 dist-test-slave-v5j9 krb5kdc[20180](info): set up 2 sockets
Sep 24 16:17:46 dist-test-slave-v5j9 krb5kdc[20180](info): commencing operation
krb5kdc: starting...
W20260924 16:17:47.757494 30632 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Sep 24 16:17:47 dist-test-slave-v5j9 krb5kdc[20180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266667, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:17:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:17:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790266323169505-30632-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790266323169505-30632-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 16:17:48.023418 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790266323169505-30632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790266323169505-30632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790266323169505-30632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44331
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790266323169505-30632-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44331
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:17:48.362819 20193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:17:48.363101 20193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:17:48.363312 20193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:17:48.370376 20193 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:17:48.370503 20193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:17:48.370607 20193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:17:48.370726 20193 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:17:48.370805 20193 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:17:48.380407 20193 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790266323169505-30632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790266323169505-30632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44331
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790266323169505-30632-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790266323169505-30632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:17:48.381740 20193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:17:48.383101 20193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:17:48.394955 20199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:48.395002 20198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:48.395466 20202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:17:48.395831 20193 server_base.cc:1034] running on GCE node
I20260924 16:17:48.396654 20193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:17:48.398335 20193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:17:48.399565 20193 hybrid_clock.cc:648] HybridClock initialized: now 1790266668399499 us; error 60 us; skew 500 ppm
I20260924 16:17:48.399897 20193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:17:48 dist-test-slave-v5j9 krb5kdc[20180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266668, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:17:48.403136 20193 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 16:17:48.404739 20193 webserver.cc:460] Webserver started at http://127.0.0.1:43185/ using document root <none> and password file <none>
I20260924 16:17:48.405514 20193 fs_manager.cc:359] Metadata directory not provided
I20260924 16:17:48.405643 20193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:17:48.405968 20193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:17:48.409004 20193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790266323169505-30632-0/minicluster-data/master-0/data/instance:
uuid: "81fc6ff8f3994ba8bd09cfc0f9bf9e21"
format_stamp: "Formatted at 2026-09-24 16:17:48 on dist-test-slave-v5j9"
server_key: "1d66135c262386833f7be58082096cb2"
server_key_iv: "5dd1dbfee676ee25d72e246984888a0e"
server_key_version: "encryptionkey@0"
I20260924 16:17:48.409787 20193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790266323169505-30632-0/minicluster-data/master-0/wal/instance:
uuid: "81fc6ff8f3994ba8bd09cfc0f9bf9e21"
format_stamp: "Formatted at 2026-09-24 16:17:48 on dist-test-slave-v5j9"
server_key: "1d66135c262386833f7be58082096cb2"
server_key_iv: "5dd1dbfee676ee25d72e246984888a0e"
server_key_version: "encryptionkey@0"
I20260924 16:17:48.414718 20193 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260924 16:17:48.418062 20209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:17:48.419373 20193 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:17:48.419549 20193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790266323169505-30632-0/minicluster-data/master-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790266323169505-30632-0/minicluster-data/master-0/wal
uuid: "81fc6ff8f3994ba8bd09cfc0f9bf9e21"
format_stamp: "Formatted at 2026-09-24 16:17:48 on dist-test-slave-v5j9"
server_key: "1d66135c262386833f7be58082096cb2"
server_key_iv: "5dd1dbfee676ee25d72e246984888a0e"
server_key_version: "encryptionkey@0"
I20260924 16:17:48.419749 20193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790266323169505-30632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790266323169505-30632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790266323169505-30632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:17:48.425554 20193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:17:48.429566 20193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:17:48.429863 20193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:17:48.448732 20193 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44331
I20260924 16:17:48.448777 20260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44331 every 8 connection(s)
I20260924 16:17:48.450294 20193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
I20260924 16:17:48.454255 20261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:17:48.459909 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 20193
I20260924 16:17:48.460108 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790266323169505-30632-0/minicluster-data/master-0/wal/instance
I20260924 16:17:48.460633 30632 external_mini_cluster.cc:1428] Setting key 374c39760c09aca91551cfaaa8234698
I20260924 16:17:48.465960 20261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81fc6ff8f3994ba8bd09cfc0f9bf9e21: Bootstrap starting.
I20260924 16:17:48.468515 20265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:48.462496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.241:35955 (local address 127.0.0.1:44331)
0924 16:17:48.463346 (+ 850us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:48.463358 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:48.463413 (+ 55us) server_negotiation.cc:407] Connection header received
0924 16:17:48.465091 (+ 1678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:48.465114 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:48.465578 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:48.465948 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:48.467005 (+ 1057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:48.467011 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:48.467024 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:17:48.467038 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:48.467094 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:17:48.467112 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:48.467289 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:48.467384 (+ 95us) server_negotiation.cc:299] Negotiation successful
0924 16:17:48.467621 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":380,"threads_started":1}
I20260924 16:17:48.469525 20261 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81fc6ff8f3994ba8bd09cfc0f9bf9e21: Neither blocks nor log segments found. Creating new log.
I20260924 16:17:48.470654 20261 log.cc:824] T 00000000000000000000000000000000 P 81fc6ff8f3994ba8bd09cfc0f9bf9e21: Log is configured to *not* fsync() on all Append() calls
I20260924 16:17:48.472348 30632 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:17:48.473546 20261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81fc6ff8f3994ba8bd09cfc0f9bf9e21: No bootstrap required, opened a new log
I20260924 16:17:48.479280 20265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:48.476729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.241:55193 (local address 127.0.0.1:44331)
0924 16:17:48.476908 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:48.476914 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:48.477384 (+ 470us) server_negotiation.cc:407] Connection header received
0924 16:17:48.477545 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:48.477550 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:48.477616 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:48.477719 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:48.478621 (+ 902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:48.478626 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:48.478631 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:17:48.478649 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:48.478692 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:17:48.478715 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:48.478829 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:48.478930 (+ 101us) server_negotiation.cc:299] Negotiation successful
0924 16:17:48.478985 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 16:17:48.481699 20261 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81fc6ff8f3994ba8bd09cfc0f9bf9e21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81fc6ff8f3994ba8bd09cfc0f9bf9e21" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44331 } }
I20260924 16:17:48.482049 20261 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81fc6ff8f3994ba8bd09cfc0f9bf9e21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:17:48.482118 20261 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81fc6ff8f3994ba8bd09cfc0f9bf9e21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81fc6ff8f3994ba8bd09cfc0f9bf9e21, State: Initialized, Role: FOLLOWER
I20260924 16:17:48.482673 20261 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81fc6ff8f3994ba8bd09cfc0f9bf9e21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81fc6ff8f3994ba8bd09cfc0f9bf9e21" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44331 } }
I20260924 16:17:48.482896 20261 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81fc6ff8f3994ba8bd09cfc0f9bf9e21 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:17:48.482987 20261 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81fc6ff8f3994ba8bd09cfc0f9bf9e21 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:17:48.483104 20261 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81fc6ff8f3994ba8bd09cfc0f9bf9e21 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:17:48.485198 20261 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81fc6ff8f3994ba8bd09cfc0f9bf9e21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81fc6ff8f3994ba8bd09cfc0f9bf9e21" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44331 } }
I20260924 16:17:48.485718 20261 leader_election.cc:302] T 00000000000000000000000000000000 P 81fc6ff8f3994ba8bd09cfc0f9bf9e21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81fc6ff8f3994ba8bd09cfc0f9bf9e21; no voters:
I20260924 16:17:48.486243 20261 leader_election.cc:288] T 00000000000000000000000000000000 P 81fc6ff8f3994ba8bd09cfc0f9bf9e21 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:17:48.486346 20272 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81fc6ff8f3994ba8bd09cfc0f9bf9e21 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:17:48.487025 20272 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81fc6ff8f3994ba8bd09cfc0f9bf9e21 [term 1 LEADER]: Becoming Leader. State: Replica: 81fc6ff8f3994ba8bd09cfc0f9bf9e21, State: Running, Role: LEADER
I20260924 16:17:48.487566 20272 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81fc6ff8f3994ba8bd09cfc0f9bf9e21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81fc6ff8f3994ba8bd09cfc0f9bf9e21" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44331 } }
I20260924 16:17:48.488282 20261 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81fc6ff8f3994ba8bd09cfc0f9bf9e21 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:17:48.490525 20274 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81fc6ff8f3994ba8bd09cfc0f9bf9e21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81fc6ff8f3994ba8bd09cfc0f9bf9e21. Latest consensus state: current_term: 1 leader_uuid: "81fc6ff8f3994ba8bd09cfc0f9bf9e21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81fc6ff8f3994ba8bd09cfc0f9bf9e21" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44331 } } }
I20260924 16:17:48.491017 20274 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81fc6ff8f3994ba8bd09cfc0f9bf9e21 [sys.catalog]: This master's current role is: LEADER
I20260924 16:17:48.490569 20273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81fc6ff8f3994ba8bd09cfc0f9bf9e21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81fc6ff8f3994ba8bd09cfc0f9bf9e21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81fc6ff8f3994ba8bd09cfc0f9bf9e21" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44331 } } }
I20260924 16:17:48.492038 20273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81fc6ff8f3994ba8bd09cfc0f9bf9e21 [sys.catalog]: This master's current role is: LEADER
I20260924 16:17:48.492698 20279 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:17:48.498909 20279 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:17:48.507141 20279 catalog_manager.cc:1347] Generated new cluster ID: b9756144a3964194b5ad876706969faa
I20260924 16:17:48.507256 20279 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:17:48.523528 20279 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:17:48.524024 20279 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:17:48.539647 20279 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81fc6ff8f3994ba8bd09cfc0f9bf9e21: Generated new TSK 0
I20260924 16:17:48.540711 20279 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:17:48.547557 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 20193
W20260924 16:17:48.548760 20256 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.241:35955
W20260924 16:17:48.549491 30632 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 20193: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 16:17:48.549635 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 20193
2026-09-24T16:17:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1912 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790266323169505-30632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:17:48 dist-test-slave-v5j9 krb5kdc[20296](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:17:48 dist-test-slave-v5j9 krb5kdc[20296](info): set up 2 sockets
Sep 24 16:17:48 dist-test-slave-v5j9 krb5kdc[20296](info): commencing operation
krb5kdc: starting...
W20260924 16:17:49.657289 30632 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.003s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Sep 24 16:17:49 dist-test-slave-v5j9 krb5kdc[20296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:17:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:17:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.234.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab.
Entry for principal kudu/127.29.234.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab.
I20260924 16:17:49.926694 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790266323169505-30632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790266323169505-30632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790266323169505-30632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:35707
--webserver_interface=127.29.234.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab
--principal=kudu/127.29.234.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:37381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.234.62:35707
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:17:50.275955 20309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:17:50.276284 20309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:17:50.276378 20309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:17:50.284032 20309 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:17:50.284190 20309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:17:50.284252 20309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:17:50.284322 20309 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:17:50.284364 20309 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:17:50.294582 20309 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:37381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790266323169505-30632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790266323169505-30632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.234.62:35707
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab
--principal=kudu/127.29.234.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:35707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.234.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790266323169505-30632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:17:50.295912 20309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:17:50.297256 20309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:17:50.310304 20314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:50.310328 20315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:50.310277 20317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:17:50.311144 20309 server_base.cc:1034] running on GCE node
I20260924 16:17:50.311975 20309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:17:50.313463 20309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:17:50.314687 20309 hybrid_clock.cc:648] HybridClock initialized: now 1790266670314634 us; error 52 us; skew 500 ppm
I20260924 16:17:50.315069 20309 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:17:50 dist-test-slave-v5j9 krb5kdc[20296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266670, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:17:50.318274 20309 init.cc:377] Logged in from keytab as kudu/127.29.234.62@KRBTEST.COM (short username kudu)
I20260924 16:17:50.319861 20309 webserver.cc:460] Webserver started at http://127.29.234.62:37775/ using document root <none> and password file <none>
I20260924 16:17:50.320587 20309 fs_manager.cc:359] Metadata directory not provided
I20260924 16:17:50.320749 20309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:17:50.321063 20309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:17:50.324210 20309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790266323169505-30632-0/minicluster-data/master-0/data/instance:
uuid: "aac82bec55fc45b8bd9e2b2df9009079"
format_stamp: "Formatted at 2026-09-24 16:17:50 on dist-test-slave-v5j9"
server_key: "c82599387cb6fcd8661dcbadc1df512a"
server_key_iv: "722c8e0f3f7933990df21011f24d9e87"
server_key_version: "encryptionkey@0"
I20260924 16:17:50.324980 20309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790266323169505-30632-0/minicluster-data/master-0/wal/instance:
uuid: "aac82bec55fc45b8bd9e2b2df9009079"
format_stamp: "Formatted at 2026-09-24 16:17:50 on dist-test-slave-v5j9"
server_key: "c82599387cb6fcd8661dcbadc1df512a"
server_key_iv: "722c8e0f3f7933990df21011f24d9e87"
server_key_version: "encryptionkey@0"
I20260924 16:17:50.330360 20309 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 16:17:50.334017 20324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:17:50.335384 20309 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:17:50.335574 20309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790266323169505-30632-0/minicluster-data/master-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790266323169505-30632-0/minicluster-data/master-0/wal
uuid: "aac82bec55fc45b8bd9e2b2df9009079"
format_stamp: "Formatted at 2026-09-24 16:17:50 on dist-test-slave-v5j9"
server_key: "c82599387cb6fcd8661dcbadc1df512a"
server_key_iv: "722c8e0f3f7933990df21011f24d9e87"
server_key_version: "encryptionkey@0"
I20260924 16:17:50.335790 20309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790266323169505-30632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790266323169505-30632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790266323169505-30632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:17:50.352551 20309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:17:50.356827 20309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:17:50.357138 20309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:17:50.375936 20309 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.62:35707
I20260924 16:17:50.375958 20376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.62:35707 every 8 connection(s)
I20260924 16:17:50.377430 20309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
I20260924 16:17:50.381210 20377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:17:50.383450 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 20309
I20260924 16:17:50.383635 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790266323169505-30632-0/minicluster-data/master-0/wal/instance
I20260924 16:17:50.384191 30632 external_mini_cluster.cc:1428] Setting key e20fb312569cd6f24c37e187ebf57b00
I20260924 16:17:50.393121 20377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aac82bec55fc45b8bd9e2b2df9009079: Bootstrap starting.
I20260924 16:17:50.396415 20377 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aac82bec55fc45b8bd9e2b2df9009079: Neither blocks nor log segments found. Creating new log.
I20260924 16:17:50.397541 20377 log.cc:824] T 00000000000000000000000000000000 P aac82bec55fc45b8bd9e2b2df9009079: Log is configured to *not* fsync() on all Append() calls
Sep 24 16:17:50 dist-test-slave-v5j9 krb5kdc[20296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:17:50.400790 20377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aac82bec55fc45b8bd9e2b2df9009079: No bootstrap required, opened a new log
I20260924 16:17:50.407224 20380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:50.386201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54819 (local address 127.29.234.62:35707)
0924 16:17:50.387059 (+ 858us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:50.387071 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:50.387122 (+ 51us) server_negotiation.cc:407] Connection header received
0924 16:17:50.388816 (+ 1694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:50.388841 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:50.389251 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:50.389706 (+ 455us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:50.391197 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:50.392527 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:50.393933 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:50.394632 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:50.394823 (+ 191us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:50.399386 (+ 4563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:50.399395 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:50.399410 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:17:50.399447 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:50.402741 (+ 3294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:17:50.403719 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:17:50.403726 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:17:50.403733 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:17:50.403914 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:17:50.404423 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:17:50.404426 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:17:50.404428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:17:50.404956 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:50.405210 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:50.405737 (+ 527us) server_negotiation.cc:299] Negotiation successful
0924 16:17:50.406083 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":333,"threads_started":1}
I20260924 16:17:50.410058 20377 raft_consensus.cc:348] T 00000000000000000000000000000000 P aac82bec55fc45b8bd9e2b2df9009079 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aac82bec55fc45b8bd9e2b2df9009079" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 35707 } }
I20260924 16:17:50.410521 20377 raft_consensus.cc:374] T 00000000000000000000000000000000 P aac82bec55fc45b8bd9e2b2df9009079 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:17:50.410670 20377 raft_consensus.cc:729] T 00000000000000000000000000000000 P aac82bec55fc45b8bd9e2b2df9009079 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aac82bec55fc45b8bd9e2b2df9009079, State: Initialized, Role: FOLLOWER
I20260924 16:17:50.411258 20377 consensus_queue.cc:260] T 00000000000000000000000000000000 P aac82bec55fc45b8bd9e2b2df9009079 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aac82bec55fc45b8bd9e2b2df9009079" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 35707 } }
I20260924 16:17:50.411458 30632 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:17:50.411553 20377 raft_consensus.cc:388] T 00000000000000000000000000000000 P aac82bec55fc45b8bd9e2b2df9009079 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:17:50.411684 20377 raft_consensus.cc:482] T 00000000000000000000000000000000 P aac82bec55fc45b8bd9e2b2df9009079 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:17:50.411840 20377 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aac82bec55fc45b8bd9e2b2df9009079 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:17:50.415169 20377 raft_consensus.cc:504] T 00000000000000000000000000000000 P aac82bec55fc45b8bd9e2b2df9009079 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aac82bec55fc45b8bd9e2b2df9009079" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 35707 } }
I20260924 16:17:50.415836 20377 leader_election.cc:302] T 00000000000000000000000000000000 P aac82bec55fc45b8bd9e2b2df9009079 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aac82bec55fc45b8bd9e2b2df9009079; no voters:
I20260924 16:17:50.416467 20377 leader_election.cc:288] T 00000000000000000000000000000000 P aac82bec55fc45b8bd9e2b2df9009079 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:17:50.416613 20386 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aac82bec55fc45b8bd9e2b2df9009079 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:17:50.418463 20386 raft_consensus.cc:686] T 00000000000000000000000000000000 P aac82bec55fc45b8bd9e2b2df9009079 [term 1 LEADER]: Becoming Leader. State: Replica: aac82bec55fc45b8bd9e2b2df9009079, State: Running, Role: LEADER
I20260924 16:17:50.418906 20377 sys_catalog.cc:564] T 00000000000000000000000000000000 P aac82bec55fc45b8bd9e2b2df9009079 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:17:50.419165 20386 consensus_queue.cc:237] T 00000000000000000000000000000000 P aac82bec55fc45b8bd9e2b2df9009079 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aac82bec55fc45b8bd9e2b2df9009079" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 35707 } }
I20260924 16:17:50.424166 20388 sys_catalog.cc:455] T 00000000000000000000000000000000 P aac82bec55fc45b8bd9e2b2df9009079 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aac82bec55fc45b8bd9e2b2df9009079. Latest consensus state: current_term: 1 leader_uuid: "aac82bec55fc45b8bd9e2b2df9009079" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aac82bec55fc45b8bd9e2b2df9009079" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 35707 } } }
I20260924 16:17:50.424530 20388 sys_catalog.cc:458] T 00000000000000000000000000000000 P aac82bec55fc45b8bd9e2b2df9009079 [sys.catalog]: This master's current role is: LEADER
I20260924 16:17:50.425083 20387 sys_catalog.cc:455] T 00000000000000000000000000000000 P aac82bec55fc45b8bd9e2b2df9009079 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aac82bec55fc45b8bd9e2b2df9009079" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aac82bec55fc45b8bd9e2b2df9009079" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 35707 } } }
I20260924 16:17:50.425493 20387 sys_catalog.cc:458] T 00000000000000000000000000000000 P aac82bec55fc45b8bd9e2b2df9009079 [sys.catalog]: This master's current role is: LEADER
I20260924 16:17:50.437796 20393 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:17:50.439941 20380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:50.416934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60243 (local address 127.29.234.62:35707)
0924 16:17:50.417210 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:50.417219 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:50.418332 (+ 1113us) server_negotiation.cc:407] Connection header received
0924 16:17:50.418777 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:50.418782 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:50.418859 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:50.418970 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:50.421034 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:50.422836 (+ 1802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:50.425686 (+ 2850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:50.426669 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:50.426833 (+ 164us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:50.431336 (+ 4503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:50.431347 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:50.431354 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:17:50.431415 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:50.435979 (+ 4564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:17:50.437146 (+ 1167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:17:50.437151 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:17:50.437156 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:17:50.437363 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:17:50.438029 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:17:50.438034 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:17:50.438037 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:17:50.438453 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:50.438726 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:50.439395 (+ 669us) server_negotiation.cc:299] Negotiation successful
0924 16:17:50.439581 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 16:17:50.445479 20393 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:17:50.456259 20393 catalog_manager.cc:1347] Generated new cluster ID: dec63ab11aa24ba186c4300882f525ef
I20260924 16:17:50.456420 20393 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:17:50.474941 20393 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:17:50.476318 20393 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:17:50.482472 20393 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aac82bec55fc45b8bd9e2b2df9009079: Generated new TSK 0
I20260924 16:17:50.483562 20393 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:17:50.496693 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 20309
I20260924 16:17:50.497996 20371 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:17:50.592386 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 20309
W20260924 16:17:50.611008 20294 connection.cc:535] client connection to 127.29.234.62:35707 recv error: Network error: failed to read from TLS socket (remote: 127.29.234.62:35707): Connection reset by peer (error 104)
2026-09-24T16:17:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2043 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790266323169505-30632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:17:50 dist-test-slave-v5j9 krb5kdc[20410](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:17:50 dist-test-slave-v5j9 krb5kdc[20410](info): set up 2 sockets
Sep 24 16:17:50 dist-test-slave-v5j9 krb5kdc[20410](info): commencing operation
krb5kdc: starting...
W20260924 16:17:51.713402 30632 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Sep 24 16:17:51 dist-test-slave-v5j9 krb5kdc[20410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:17:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:17:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.234.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab.
Entry for principal kudu/127.29.234.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab.
I20260924 16:17:51.964504 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790266323169505-30632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790266323169505-30632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790266323169505-30632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:38851
--webserver_interface=127.29.234.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab
--principal=kudu/127.29.234.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:33729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.234.62:38851
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:17:52.313918 20423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:17:52.314204 20423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:17:52.314297 20423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:17:52.321527 20423 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:17:52.321650 20423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:17:52.321696 20423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:17:52.321758 20423 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:17:52.321810 20423 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:17:52.332366 20423 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:33729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790266323169505-30632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790266323169505-30632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.234.62:38851
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab
--principal=kudu/127.29.234.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:38851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.234.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790266323169505-30632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:17:52.333751 20423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:17:52.335078 20423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:17:52.347002 20428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:52.347002 20431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:52.347043 20429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:17:52.347862 20423 server_base.cc:1034] running on GCE node
I20260924 16:17:52.348647 20423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:17:52.350061 20423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:17:52.351262 20423 hybrid_clock.cc:648] HybridClock initialized: now 1790266672351219 us; error 32 us; skew 500 ppm
I20260924 16:17:52.351613 20423 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:17:52 dist-test-slave-v5j9 krb5kdc[20410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266672, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:17:52.368517 20423 init.cc:377] Logged in from keytab as kudu/127.29.234.62@KRBTEST.COM (short username kudu)
I20260924 16:17:52.370708 20423 webserver.cc:460] Webserver started at http://127.29.234.62:36387/ using document root <none> and password file <none>
I20260924 16:17:52.371534 20423 fs_manager.cc:359] Metadata directory not provided
I20260924 16:17:52.371686 20423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:17:52.372032 20423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:17:52.375245 20423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790266323169505-30632-0/minicluster-data/master-0/data/instance:
uuid: "33376757d6694d34b87325a4672d5b43"
format_stamp: "Formatted at 2026-09-24 16:17:52 on dist-test-slave-v5j9"
server_key: "8d2ed8b12018c1aebffc58c65e4791fc"
server_key_iv: "0e0fcbc2971b4e01f86be421be4fb2ca"
server_key_version: "encryptionkey@0"
I20260924 16:17:52.376041 20423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790266323169505-30632-0/minicluster-data/master-0/wal/instance:
uuid: "33376757d6694d34b87325a4672d5b43"
format_stamp: "Formatted at 2026-09-24 16:17:52 on dist-test-slave-v5j9"
server_key: "8d2ed8b12018c1aebffc58c65e4791fc"
server_key_iv: "0e0fcbc2971b4e01f86be421be4fb2ca"
server_key_version: "encryptionkey@0"
I20260924 16:17:52.381590 20423 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 16:17:52.385221 20438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:17:52.386656 20423 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:17:52.386862 20423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790266323169505-30632-0/minicluster-data/master-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790266323169505-30632-0/minicluster-data/master-0/wal
uuid: "33376757d6694d34b87325a4672d5b43"
format_stamp: "Formatted at 2026-09-24 16:17:52 on dist-test-slave-v5j9"
server_key: "8d2ed8b12018c1aebffc58c65e4791fc"
server_key_iv: "0e0fcbc2971b4e01f86be421be4fb2ca"
server_key_version: "encryptionkey@0"
I20260924 16:17:52.387069 20423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790266323169505-30632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790266323169505-30632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790266323169505-30632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:17:52.401544 20423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:17:52.408679 20423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:17:52.409005 20423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:17:52.428380 20423 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.62:38851
I20260924 16:17:52.428372 20491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.62:38851 every 8 connection(s)
I20260924 16:17:52.429929 20423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
I20260924 16:17:52.430739 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 20423
I20260924 16:17:52.430962 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790266323169505-30632-0/minicluster-data/master-0/wal/instance
I20260924 16:17:52.431494 30632 external_mini_cluster.cc:1428] Setting key a704f29b0a32eb8495d672ec746dbbd6
I20260924 16:17:52.434257 20492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:17:52.445747 20495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:52.433440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34045 (local address 127.29.234.62:38851)
0924 16:17:52.434238 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:52.434255 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:52.434299 (+ 44us) server_negotiation.cc:407] Connection header received
0924 16:17:52.436013 (+ 1714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:52.436037 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:52.436441 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:52.436869 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:52.437515 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:52.438833 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:52.440169 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:52.440847 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:52.441042 (+ 195us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:52.443311 (+ 2269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:52.443319 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:52.443336 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:17:52.443923 (+ 587us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:52.444001 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:17:52.444026 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:52.444231 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:52.444508 (+ 277us) server_negotiation.cc:299] Negotiation successful
0924 16:17:52.444698 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":355,"threads_started":1}
I20260924 16:17:52.446005 20492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33376757d6694d34b87325a4672d5b43: Bootstrap starting.
I20260924 16:17:52.450037 20492 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33376757d6694d34b87325a4672d5b43: Neither blocks nor log segments found. Creating new log.
I20260924 16:17:52.450066 30632 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:17:52.451054 20492 log.cc:824] T 00000000000000000000000000000000 P 33376757d6694d34b87325a4672d5b43: Log is configured to *not* fsync() on all Append() calls
I20260924 16:17:52.454083 20492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33376757d6694d34b87325a4672d5b43: No bootstrap required, opened a new log
I20260924 16:17:52.461840 20495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:52.454399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43117 (local address 127.29.234.62:38851)
0924 16:17:52.454574 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:52.454578 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:52.455214 (+ 636us) server_negotiation.cc:407] Connection header received
0924 16:17:52.455432 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:52.455437 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:52.455501 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:52.455606 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:52.456119 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:52.456925 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:52.458326 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:52.458829 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:52.458978 (+ 149us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:52.460991 (+ 2013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:52.460999 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:52.461002 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:17:52.461023 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:52.461065 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:17:52.461084 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:52.461227 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:52.461489 (+ 262us) server_negotiation.cc:299] Negotiation successful
0924 16:17:52.461545 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 16:17:52.462998 20492 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33376757d6694d34b87325a4672d5b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33376757d6694d34b87325a4672d5b43" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 38851 } }
I20260924 16:17:52.463536 20492 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33376757d6694d34b87325a4672d5b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:17:52.463660 20492 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33376757d6694d34b87325a4672d5b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33376757d6694d34b87325a4672d5b43, State: Initialized, Role: FOLLOWER
I20260924 16:17:52.464289 20492 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33376757d6694d34b87325a4672d5b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33376757d6694d34b87325a4672d5b43" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 38851 } }
I20260924 16:17:52.464505 20492 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33376757d6694d34b87325a4672d5b43 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:17:52.464608 20492 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33376757d6694d34b87325a4672d5b43 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:17:52.464744 20492 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33376757d6694d34b87325a4672d5b43 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:17:52.467284 20492 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33376757d6694d34b87325a4672d5b43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33376757d6694d34b87325a4672d5b43" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 38851 } }
I20260924 16:17:52.467911 20492 leader_election.cc:302] T 00000000000000000000000000000000 P 33376757d6694d34b87325a4672d5b43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33376757d6694d34b87325a4672d5b43; no voters:
I20260924 16:17:52.468497 20492 leader_election.cc:288] T 00000000000000000000000000000000 P 33376757d6694d34b87325a4672d5b43 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:17:52.468686 20502 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33376757d6694d34b87325a4672d5b43 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:17:52.469537 20502 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33376757d6694d34b87325a4672d5b43 [term 1 LEADER]: Becoming Leader. State: Replica: 33376757d6694d34b87325a4672d5b43, State: Running, Role: LEADER
I20260924 16:17:52.470180 20502 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33376757d6694d34b87325a4672d5b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33376757d6694d34b87325a4672d5b43" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 38851 } }
I20260924 16:17:52.470920 20492 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33376757d6694d34b87325a4672d5b43 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:17:52.473613 20504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33376757d6694d34b87325a4672d5b43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33376757d6694d34b87325a4672d5b43. Latest consensus state: current_term: 1 leader_uuid: "33376757d6694d34b87325a4672d5b43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33376757d6694d34b87325a4672d5b43" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 38851 } } }
I20260924 16:17:52.474213 20503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33376757d6694d34b87325a4672d5b43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33376757d6694d34b87325a4672d5b43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33376757d6694d34b87325a4672d5b43" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 38851 } } }
I20260924 16:17:52.474560 20504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33376757d6694d34b87325a4672d5b43 [sys.catalog]: This master's current role is: LEADER
I20260924 16:17:52.474615 20503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33376757d6694d34b87325a4672d5b43 [sys.catalog]: This master's current role is: LEADER
I20260924 16:17:52.482503 20511 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:17:52.487854 20511 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:17:52.496958 20511 catalog_manager.cc:1347] Generated new cluster ID: 4ca5c0e02b084a1290034c750c6a45fe
I20260924 16:17:52.497079 20511 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:17:52.516871 20511 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:17:52.518339 20511 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:17:52.526417 20511 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33376757d6694d34b87325a4672d5b43: Generated new TSK 0
I20260924 16:17:52.527282 20511 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:17:52.541462 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 20423
W20260924 16:17:52.543229 20487 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:34045
W20260924 16:17:52.543987 30632 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 20423: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 16:17:52.544145 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 20423
W20260924 16:17:52.562071 20408 connection.cc:535] client connection to 127.29.234.62:38851 recv error: Network error: failed to read from TLS socket (remote: 127.29.234.62:38851): Connection reset by peer (error 104)
2026-09-24T16:17:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1951 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790266323169505-30632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:17:52 dist-test-slave-v5j9 krb5kdc[20523](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:17:52 dist-test-slave-v5j9 krb5kdc[20523](info): set up 2 sockets
Sep 24 16:17:52 dist-test-slave-v5j9 krb5kdc[20523](info): commencing operation
krb5kdc: starting...
W20260924 16:17:53.657588 30632 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Sep 24 16:17:53 dist-test-slave-v5j9 krb5kdc[20523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:17:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:17:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.234.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab.
Entry for principal kudu/127.29.234.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab.
I20260924 16:17:53.924897 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790266323169505-30632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790266323169505-30632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790266323169505-30632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:33203
--webserver_interface=127.29.234.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab
--principal=kudu/127.29.234.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:39109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.234.62:33203
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:17:54.270751 20537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:17:54.271075 20537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:17:54.271247 20537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:17:54.278465 20537 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:17:54.278610 20537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:17:54.278709 20537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:17:54.278810 20537 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:17:54.278884 20537 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:17:54.288895 20537 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:39109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790266323169505-30632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790266323169505-30632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.234.62:33203
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab
--principal=kudu/127.29.234.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:33203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.234.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790266323169505-30632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:17:54.290340 20537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:17:54.291769 20537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:17:54.304423 20543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:54.304467 20546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:54.304508 20544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:17:54.305429 20537 server_base.cc:1034] running on GCE node
I20260924 16:17:54.306257 20537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:17:54.307705 20537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:17:54.309020 20537 hybrid_clock.cc:648] HybridClock initialized: now 1790266674308918 us; error 77 us; skew 500 ppm
I20260924 16:17:54.309486 20537 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:17:54 dist-test-slave-v5j9 krb5kdc[20523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266674, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:17:54.313812 20537 init.cc:377] Logged in from keytab as kudu/127.29.234.62@KRBTEST.COM (short username kudu)
I20260924 16:17:54.315353 20537 webserver.cc:460] Webserver started at http://127.29.234.62:43853/ using document root <none> and password file <none>
I20260924 16:17:54.316074 20537 fs_manager.cc:359] Metadata directory not provided
I20260924 16:17:54.316231 20537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:17:54.316547 20537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:17:54.319597 20537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790266323169505-30632-0/minicluster-data/master-0/data/instance:
uuid: "4e48e1471dfe47a49d87e3b5c7084973"
format_stamp: "Formatted at 2026-09-24 16:17:54 on dist-test-slave-v5j9"
server_key: "8cdefa128ef10a9ec7a68dbd25926908"
server_key_iv: "3f784880547849422dc195b13c72c3bc"
server_key_version: "encryptionkey@0"
I20260924 16:17:54.320474 20537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790266323169505-30632-0/minicluster-data/master-0/wal/instance:
uuid: "4e48e1471dfe47a49d87e3b5c7084973"
format_stamp: "Formatted at 2026-09-24 16:17:54 on dist-test-slave-v5j9"
server_key: "8cdefa128ef10a9ec7a68dbd25926908"
server_key_iv: "3f784880547849422dc195b13c72c3bc"
server_key_version: "encryptionkey@0"
I20260924 16:17:54.325955 20537 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 16:17:54.329758 20553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:17:54.331250 20537 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 16:17:54.331461 20537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790266323169505-30632-0/minicluster-data/master-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790266323169505-30632-0/minicluster-data/master-0/wal
uuid: "4e48e1471dfe47a49d87e3b5c7084973"
format_stamp: "Formatted at 2026-09-24 16:17:54 on dist-test-slave-v5j9"
server_key: "8cdefa128ef10a9ec7a68dbd25926908"
server_key_iv: "3f784880547849422dc195b13c72c3bc"
server_key_version: "encryptionkey@0"
I20260924 16:17:54.331684 20537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790266323169505-30632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790266323169505-30632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790266323169505-30632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:17:54.337641 20537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:17:54.344730 20537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:17:54.345029 20537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:17:54.363085 20537 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.62:33203
I20260924 16:17:54.363107 20605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.62:33203 every 8 connection(s)
I20260924 16:17:54.364449 20537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
I20260924 16:17:54.368387 20607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:17:54.371300 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 20537
I20260924 16:17:54.371505 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790266323169505-30632-0/minicluster-data/master-0/wal/instance
I20260924 16:17:54.371961 30632 external_mini_cluster.cc:1428] Setting key a6f4d038a4db20b4ed8ca7970fb84322
I20260924 16:17:54.382035 20607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e48e1471dfe47a49d87e3b5c7084973: Bootstrap starting.
I20260924 16:17:54.382447 20610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:54.373983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37365 (local address 127.29.234.62:33203)
0924 16:17:54.374989 (+ 1006us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:54.375005 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:54.375057 (+ 52us) server_negotiation.cc:407] Connection header received
0924 16:17:54.377415 (+ 2358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:54.377452 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:54.377949 (+ 497us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:54.378448 (+ 499us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:54.380322 (+ 1874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:54.380331 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:54.380349 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:17:54.380376 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:54.380460 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:17:54.380494 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:54.380771 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:54.380908 (+ 137us) server_negotiation.cc:299] Negotiation successful
0924 16:17:54.381113 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":757,"thread_start_us":370,"threads_started":1}
I20260924 16:17:54.386250 20607 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e48e1471dfe47a49d87e3b5c7084973: Neither blocks nor log segments found. Creating new log.
I20260924 16:17:54.386922 30632 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:17:54.387532 20607 log.cc:824] T 00000000000000000000000000000000 P 4e48e1471dfe47a49d87e3b5c7084973: Log is configured to *not* fsync() on all Append() calls
I20260924 16:17:54.390914 20607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e48e1471dfe47a49d87e3b5c7084973: No bootstrap required, opened a new log
I20260924 16:17:54.395557 20610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:54.391969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47709 (local address 127.29.234.62:33203)
0924 16:17:54.392238 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:54.392247 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:54.392599 (+ 352us) server_negotiation.cc:407] Connection header received
0924 16:17:54.392818 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:54.392828 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:54.392892 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:54.393009 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:54.394847 (+ 1838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:54.394853 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:54.394858 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:17:54.394900 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:54.394942 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:17:54.394966 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:54.395115 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:54.395197 (+ 82us) server_negotiation.cc:299] Negotiation successful
0924 16:17:54.395269 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 16:17:54.400663 20607 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e48e1471dfe47a49d87e3b5c7084973 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e48e1471dfe47a49d87e3b5c7084973" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 33203 } }
I20260924 16:17:54.401103 20607 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e48e1471dfe47a49d87e3b5c7084973 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:17:54.401224 20607 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e48e1471dfe47a49d87e3b5c7084973 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e48e1471dfe47a49d87e3b5c7084973, State: Initialized, Role: FOLLOWER
I20260924 16:17:54.401922 20607 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e48e1471dfe47a49d87e3b5c7084973 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e48e1471dfe47a49d87e3b5c7084973" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 33203 } }
I20260924 16:17:54.402177 20607 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e48e1471dfe47a49d87e3b5c7084973 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:17:54.402333 20607 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e48e1471dfe47a49d87e3b5c7084973 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:17:54.402504 20607 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e48e1471dfe47a49d87e3b5c7084973 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:17:54.404911 20607 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e48e1471dfe47a49d87e3b5c7084973 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e48e1471dfe47a49d87e3b5c7084973" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 33203 } }
I20260924 16:17:54.405552 20607 leader_election.cc:302] T 00000000000000000000000000000000 P 4e48e1471dfe47a49d87e3b5c7084973 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e48e1471dfe47a49d87e3b5c7084973; no voters:
I20260924 16:17:54.406674 20607 leader_election.cc:288] T 00000000000000000000000000000000 P 4e48e1471dfe47a49d87e3b5c7084973 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:17:54.406770 20617 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e48e1471dfe47a49d87e3b5c7084973 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:17:54.407485 20617 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e48e1471dfe47a49d87e3b5c7084973 [term 1 LEADER]: Becoming Leader. State: Replica: 4e48e1471dfe47a49d87e3b5c7084973, State: Running, Role: LEADER
I20260924 16:17:54.408118 20617 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e48e1471dfe47a49d87e3b5c7084973 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e48e1471dfe47a49d87e3b5c7084973" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 33203 } }
I20260924 16:17:54.408990 20607 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e48e1471dfe47a49d87e3b5c7084973 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:17:54.411293 20618 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e48e1471dfe47a49d87e3b5c7084973 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e48e1471dfe47a49d87e3b5c7084973" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e48e1471dfe47a49d87e3b5c7084973" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 33203 } } }
I20260924 16:17:54.412091 20618 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e48e1471dfe47a49d87e3b5c7084973 [sys.catalog]: This master's current role is: LEADER
I20260924 16:17:54.411197 20619 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e48e1471dfe47a49d87e3b5c7084973 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e48e1471dfe47a49d87e3b5c7084973. Latest consensus state: current_term: 1 leader_uuid: "4e48e1471dfe47a49d87e3b5c7084973" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e48e1471dfe47a49d87e3b5c7084973" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 33203 } } }
I20260924 16:17:54.412921 20619 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e48e1471dfe47a49d87e3b5c7084973 [sys.catalog]: This master's current role is: LEADER
I20260924 16:17:54.419011 20624 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:17:54.424757 20624 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:17:54.435359 20624 catalog_manager.cc:1347] Generated new cluster ID: a61d64a85e624a4883ce70f4aebda4e4
I20260924 16:17:54.435498 20624 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:17:54.449261 20624 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:17:54.449913 20624 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:17:54.467828 20624 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e48e1471dfe47a49d87e3b5c7084973: Generated new TSK 0
I20260924 16:17:54.469009 20624 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:17:54.480540 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 20537
W20260924 16:17:54.482173 20601 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37365
W20260924 16:17:54.482880 30632 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 20537: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 16:17:54.483042 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 20537
2026-09-24T16:17:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1938 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790266323169505-30632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:17:54 dist-test-slave-v5j9 krb5kdc[20641](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:17:54 dist-test-slave-v5j9 krb5kdc[20641](info): set up 2 sockets
Sep 24 16:17:54 dist-test-slave-v5j9 krb5kdc[20641](info): commencing operation
krb5kdc: starting...
W20260924 16:17:55.606429 30632 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Sep 24 16:17:55 dist-test-slave-v5j9 krb5kdc[20641](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266675, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:17:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:17:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.234.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab.
Entry for principal kudu/127.29.234.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab.
I20260924 16:17:55.886859 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790266323169505-30632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790266323169505-30632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790266323169505-30632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:36821
--webserver_interface=127.29.234.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab
--principal=kudu/127.29.234.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:42285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.234.62:36821
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:17:56.236608 20654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:17:56.236946 20654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:17:56.237133 20654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:17:56.244513 20654 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:17:56.244644 20654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:17:56.244765 20654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:17:56.244872 20654 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:17:56.244951 20654 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:17:56.255226 20654 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:42285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790266323169505-30632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790266323169505-30632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.234.62:36821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab
--principal=kudu/127.29.234.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:36821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.234.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790266323169505-30632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:17:56.256659 20654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:17:56.258258 20654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:17:56.274461 20659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:56.274487 20660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:56.274432 20662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:17:56.274766 20654 server_base.cc:1034] running on GCE node
I20260924 16:17:56.275863 20654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:17:56.277405 20654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:17:56.278662 20654 hybrid_clock.cc:648] HybridClock initialized: now 1790266676278593 us; error 57 us; skew 500 ppm
I20260924 16:17:56.279031 20654 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:17:56 dist-test-slave-v5j9 krb5kdc[20641](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266676, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:17:56.283548 20654 init.cc:377] Logged in from keytab as kudu/127.29.234.62@KRBTEST.COM (short username kudu)
I20260924 16:17:56.285166 20654 webserver.cc:460] Webserver started at http://127.29.234.62:36689/ using document root <none> and password file <none>
I20260924 16:17:56.286083 20654 fs_manager.cc:359] Metadata directory not provided
I20260924 16:17:56.286273 20654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:17:56.286621 20654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:17:56.289841 20654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790266323169505-30632-0/minicluster-data/master-0/data/instance:
uuid: "881cba7ad077452d904d72a1046d1af8"
format_stamp: "Formatted at 2026-09-24 16:17:56 on dist-test-slave-v5j9"
server_key: "e7e0bc22fdf3abcc276c6714a1811a01"
server_key_iv: "2fb3ec7984344f534d27cb0bbde6f3b3"
server_key_version: "encryptionkey@0"
I20260924 16:17:56.290611 20654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790266323169505-30632-0/minicluster-data/master-0/wal/instance:
uuid: "881cba7ad077452d904d72a1046d1af8"
format_stamp: "Formatted at 2026-09-24 16:17:56 on dist-test-slave-v5j9"
server_key: "e7e0bc22fdf3abcc276c6714a1811a01"
server_key_iv: "2fb3ec7984344f534d27cb0bbde6f3b3"
server_key_version: "encryptionkey@0"
I20260924 16:17:56.296134 20654 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260924 16:17:56.299830 20669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:17:56.301476 20654 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:17:56.301651 20654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790266323169505-30632-0/minicluster-data/master-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790266323169505-30632-0/minicluster-data/master-0/wal
uuid: "881cba7ad077452d904d72a1046d1af8"
format_stamp: "Formatted at 2026-09-24 16:17:56 on dist-test-slave-v5j9"
server_key: "e7e0bc22fdf3abcc276c6714a1811a01"
server_key_iv: "2fb3ec7984344f534d27cb0bbde6f3b3"
server_key_version: "encryptionkey@0"
I20260924 16:17:56.301846 20654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790266323169505-30632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790266323169505-30632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790266323169505-30632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:17:56.314349 20654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:17:56.321645 20654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:17:56.321907 20654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:17:56.340341 20654 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.62:36821
I20260924 16:17:56.340348 20723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.62:36821 every 8 connection(s)
I20260924 16:17:56.341969 20654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
I20260924 16:17:56.344702 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 20654
I20260924 16:17:56.344878 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790266323169505-30632-0/minicluster-data/master-0/wal/instance
I20260924 16:17:56.345510 30632 external_mini_cluster.cc:1428] Setting key cdca9608d7d981e60d464d3e8bab302b
I20260924 16:17:56.346441 20724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:17:56.360381 20724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 881cba7ad077452d904d72a1046d1af8: Bootstrap starting.
Sep 24 16:17:56 dist-test-slave-v5j9 krb5kdc[20641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266675, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:17:56.364418 20724 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 881cba7ad077452d904d72a1046d1af8: Neither blocks nor log segments found. Creating new log.
I20260924 16:17:56.365912 20724 log.cc:824] T 00000000000000000000000000000000 P 881cba7ad077452d904d72a1046d1af8: Log is configured to *not* fsync() on all Append() calls
I20260924 16:17:56.369756 20724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 881cba7ad077452d904d72a1046d1af8: No bootstrap required, opened a new log
I20260924 16:17:56.374083 20727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:56.347540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41577 (local address 127.29.234.62:36821)
0924 16:17:56.348363 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:56.348382 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:56.348434 (+ 52us) server_negotiation.cc:407] Connection header received
0924 16:17:56.350212 (+ 1778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:56.350242 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:56.350793 (+ 551us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:56.351250 (+ 457us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:56.352533 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:56.354624 (+ 2091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:56.355922 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:56.356992 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:56.357191 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:56.362110 (+ 4919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:56.362122 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:56.362153 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:17:56.362215 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:56.368932 (+ 6717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:17:56.369823 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:17:56.369833 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:17:56.369843 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:17:56.370063 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:17:56.370455 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:17:56.370460 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:17:56.370463 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:17:56.371236 (+ 773us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:56.371573 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:56.371924 (+ 351us) server_negotiation.cc:299] Negotiation successful
0924 16:17:56.372530 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":349,"threads_started":1}
I20260924 16:17:56.378800 30632 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:17:56.380167 20724 raft_consensus.cc:348] T 00000000000000000000000000000000 P 881cba7ad077452d904d72a1046d1af8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "881cba7ad077452d904d72a1046d1af8" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 36821 } }
I20260924 16:17:56.380605 20724 raft_consensus.cc:374] T 00000000000000000000000000000000 P 881cba7ad077452d904d72a1046d1af8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:17:56.380740 20724 raft_consensus.cc:729] T 00000000000000000000000000000000 P 881cba7ad077452d904d72a1046d1af8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 881cba7ad077452d904d72a1046d1af8, State: Initialized, Role: FOLLOWER
I20260924 16:17:56.381525 20724 consensus_queue.cc:260] T 00000000000000000000000000000000 P 881cba7ad077452d904d72a1046d1af8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "881cba7ad077452d904d72a1046d1af8" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 36821 } }
I20260924 16:17:56.381783 20724 raft_consensus.cc:388] T 00000000000000000000000000000000 P 881cba7ad077452d904d72a1046d1af8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:17:56.381927 20724 raft_consensus.cc:482] T 00000000000000000000000000000000 P 881cba7ad077452d904d72a1046d1af8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:17:56.382159 20724 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 881cba7ad077452d904d72a1046d1af8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:17:56.385001 20724 raft_consensus.cc:504] T 00000000000000000000000000000000 P 881cba7ad077452d904d72a1046d1af8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "881cba7ad077452d904d72a1046d1af8" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 36821 } }
I20260924 16:17:56.385690 20724 leader_election.cc:302] T 00000000000000000000000000000000 P 881cba7ad077452d904d72a1046d1af8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 881cba7ad077452d904d72a1046d1af8; no voters:
I20260924 16:17:56.386425 20724 leader_election.cc:288] T 00000000000000000000000000000000 P 881cba7ad077452d904d72a1046d1af8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:17:56.386531 20734 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 881cba7ad077452d904d72a1046d1af8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:17:56.387769 20734 raft_consensus.cc:686] T 00000000000000000000000000000000 P 881cba7ad077452d904d72a1046d1af8 [term 1 LEADER]: Becoming Leader. State: Replica: 881cba7ad077452d904d72a1046d1af8, State: Running, Role: LEADER
I20260924 16:17:56.388408 20734 consensus_queue.cc:237] T 00000000000000000000000000000000 P 881cba7ad077452d904d72a1046d1af8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "881cba7ad077452d904d72a1046d1af8" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 36821 } }
I20260924 16:17:56.389163 20724 sys_catalog.cc:564] T 00000000000000000000000000000000 P 881cba7ad077452d904d72a1046d1af8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:17:56.392680 20736 sys_catalog.cc:455] T 00000000000000000000000000000000 P 881cba7ad077452d904d72a1046d1af8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 881cba7ad077452d904d72a1046d1af8. Latest consensus state: current_term: 1 leader_uuid: "881cba7ad077452d904d72a1046d1af8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "881cba7ad077452d904d72a1046d1af8" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 36821 } } }
I20260924 16:17:56.393486 20735 sys_catalog.cc:455] T 00000000000000000000000000000000 P 881cba7ad077452d904d72a1046d1af8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "881cba7ad077452d904d72a1046d1af8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "881cba7ad077452d904d72a1046d1af8" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 36821 } } }
I20260924 16:17:56.394251 20736 sys_catalog.cc:458] T 00000000000000000000000000000000 P 881cba7ad077452d904d72a1046d1af8 [sys.catalog]: This master's current role is: LEADER
I20260924 16:17:56.394364 20735 sys_catalog.cc:458] T 00000000000000000000000000000000 P 881cba7ad077452d904d72a1046d1af8 [sys.catalog]: This master's current role is: LEADER
I20260924 16:17:56.395370 20742 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:17:56.402874 20742 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:17:56.405860 20727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:56.384109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36243 (local address 127.29.234.62:36821)
0924 16:17:56.384377 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:56.384384 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:56.384797 (+ 413us) server_negotiation.cc:407] Connection header received
0924 16:17:56.385035 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:56.385048 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:56.385150 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:56.385283 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:56.387318 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:56.388504 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:56.389961 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:56.390751 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:56.391589 (+ 838us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:56.395283 (+ 3694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:56.395293 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:56.395299 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:17:56.395347 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:56.401476 (+ 6129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:17:56.402451 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:17:56.402457 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:17:56.402460 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:17:56.402624 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:17:56.403119 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:17:56.403124 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:17:56.403127 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:17:56.403523 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:56.403837 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:56.405121 (+ 1284us) server_negotiation.cc:299] Negotiation successful
0924 16:17:56.405379 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":86}
I20260924 16:17:56.419976 20742 catalog_manager.cc:1347] Generated new cluster ID: ceda2037f262495c8dcfb908a1b7cb9a
I20260924 16:17:56.420104 20742 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:17:56.433534 20742 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:17:56.435514 20742 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:17:56.449208 20742 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 881cba7ad077452d904d72a1046d1af8: Generated new TSK 0
I20260924 16:17:56.450405 20742 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:17:56.463609 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 20654
I20260924 16:17:56.464813 20719 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:17:56.559686 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 20654
2026-09-24T16:17:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2077 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790266323169505-30632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:17:56 dist-test-slave-v5j9 krb5kdc[20756](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:17:56 dist-test-slave-v5j9 krb5kdc[20756](info): set up 2 sockets
Sep 24 16:17:56 dist-test-slave-v5j9 krb5kdc[20756](info): commencing operation
krb5kdc: starting...
W20260924 16:17:57.685742 30632 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Sep 24 16:17:57 dist-test-slave-v5j9 krb5kdc[20756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266677, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:17:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:17:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.234.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab.
Entry for principal kudu/127.29.234.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab.
I20260924 16:17:57.956648 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790266323169505-30632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790266323169505-30632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790266323169505-30632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:32969
--webserver_interface=127.29.234.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab
--principal=kudu/127.29.234.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:37423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.234.62:32969
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:17:58.296408 20769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:17:58.296720 20769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:17:58.296895 20769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:17:58.304404 20769 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:17:58.304533 20769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:17:58.304620 20769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:17:58.304729 20769 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:17:58.304803 20769 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:17:58.315038 20769 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:37423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790266323169505-30632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790266323169505-30632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.234.62:32969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab
--principal=kudu/127.29.234.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:32969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.234.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790266323169505-30632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:17:58.316493 20769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:17:58.317924 20769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:17:58.330009 20775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:58.330097 20774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:17:58.330097 20777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:17:58.330942 20769 server_base.cc:1034] running on GCE node
I20260924 16:17:58.331698 20769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:17:58.333035 20769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:17:58.334232 20769 hybrid_clock.cc:648] HybridClock initialized: now 1790266678334195 us; error 26 us; skew 500 ppm
I20260924 16:17:58.334599 20769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:17:58 dist-test-slave-v5j9 krb5kdc[20756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266678, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:17:58.342512 20769 init.cc:377] Logged in from keytab as kudu/127.29.234.62@KRBTEST.COM (short username kudu)
I20260924 16:17:58.344146 20769 webserver.cc:460] Webserver started at http://127.29.234.62:44837/ using document root <none> and password file <none>
I20260924 16:17:58.344837 20769 fs_manager.cc:359] Metadata directory not provided
I20260924 16:17:58.344965 20769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:17:58.345280 20769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:17:58.348366 20769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790266323169505-30632-0/minicluster-data/master-0/data/instance:
uuid: "d84cd04486824a0c90066cd6e5708eb7"
format_stamp: "Formatted at 2026-09-24 16:17:58 on dist-test-slave-v5j9"
server_key: "077c70eb215dee0c1bf753e6e8519b3f"
server_key_iv: "3aa92016f36f8232d2960a69ba7b4fc7"
server_key_version: "encryptionkey@0"
I20260924 16:17:58.349162 20769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790266323169505-30632-0/minicluster-data/master-0/wal/instance:
uuid: "d84cd04486824a0c90066cd6e5708eb7"
format_stamp: "Formatted at 2026-09-24 16:17:58 on dist-test-slave-v5j9"
server_key: "077c70eb215dee0c1bf753e6e8519b3f"
server_key_iv: "3aa92016f36f8232d2960a69ba7b4fc7"
server_key_version: "encryptionkey@0"
I20260924 16:17:58.354468 20769 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260924 16:17:58.358182 20784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:17:58.359580 20769 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:17:58.359758 20769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790266323169505-30632-0/minicluster-data/master-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790266323169505-30632-0/minicluster-data/master-0/wal
uuid: "d84cd04486824a0c90066cd6e5708eb7"
format_stamp: "Formatted at 2026-09-24 16:17:58 on dist-test-slave-v5j9"
server_key: "077c70eb215dee0c1bf753e6e8519b3f"
server_key_iv: "3aa92016f36f8232d2960a69ba7b4fc7"
server_key_version: "encryptionkey@0"
I20260924 16:17:58.359956 20769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790266323169505-30632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790266323169505-30632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790266323169505-30632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:17:58.386370 20769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:17:58.390750 20769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:17:58.391052 20769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:17:58.410164 20769 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.62:32969
I20260924 16:17:58.410209 20837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.62:32969 every 8 connection(s)
I20260924 16:17:58.411633 20769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
I20260924 16:17:58.413329 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 20769
I20260924 16:17:58.413553 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790266323169505-30632-0/minicluster-data/master-0/wal/instance
I20260924 16:17:58.413998 30632 external_mini_cluster.cc:1428] Setting key 2d565ac10b77c42631dd79ccc27bb115
I20260924 16:17:58.416138 20838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:17:58.430167 20841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:58.416001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55535 (local address 127.29.234.62:32969)
0924 16:17:58.416932 (+ 931us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:58.416948 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:58.417002 (+ 54us) server_negotiation.cc:407] Connection header received
0924 16:17:58.419423 (+ 2421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:58.419457 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:58.420142 (+ 685us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:58.420640 (+ 498us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:58.421221 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:58.423362 (+ 2141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:58.424689 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:58.425788 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:58.425986 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:58.427222 (+ 1236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:58.427233 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:58.427260 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:17:58.428038 (+ 778us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:58.428151 (+ 113us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:17:58.428184 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:58.428292 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:58.428521 (+ 229us) server_negotiation.cc:299] Negotiation successful
0924 16:17:58.428805 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":684,"thread_start_us":407,"threads_started":1}
I20260924 16:17:58.431336 20838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d84cd04486824a0c90066cd6e5708eb7: Bootstrap starting.
I20260924 16:17:58.433822 30632 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:17:58.434651 20838 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d84cd04486824a0c90066cd6e5708eb7: Neither blocks nor log segments found. Creating new log.
I20260924 16:17:58.435653 20838 log.cc:824] T 00000000000000000000000000000000 P d84cd04486824a0c90066cd6e5708eb7: Log is configured to *not* fsync() on all Append() calls
I20260924 16:17:58.439146 20838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d84cd04486824a0c90066cd6e5708eb7: No bootstrap required, opened a new log
I20260924 16:17:58.444150 20841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:17:58.438258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33517 (local address 127.29.234.62:32969)
0924 16:17:58.438414 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 16:17:58.438421 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:17:58.438841 (+ 420us) server_negotiation.cc:407] Connection header received
0924 16:17:58.439049 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:17:58.439053 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:17:58.439110 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:17:58.439211 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:17:58.439702 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:58.440403 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:58.441763 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:17:58.442234 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:17:58.442381 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:17:58.443449 (+ 1068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:17:58.443453 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:17:58.443456 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:17:58.443472 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:17:58.443511 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:17:58.443525 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:17:58.443619 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 16:17:58.443756 (+ 137us) server_negotiation.cc:299] Negotiation successful
0924 16:17:58.443846 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 16:17:58.447201 20838 raft_consensus.cc:348] T 00000000000000000000000000000000 P d84cd04486824a0c90066cd6e5708eb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d84cd04486824a0c90066cd6e5708eb7" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 32969 } }
I20260924 16:17:58.447585 20838 raft_consensus.cc:374] T 00000000000000000000000000000000 P d84cd04486824a0c90066cd6e5708eb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:17:58.447695 20838 raft_consensus.cc:729] T 00000000000000000000000000000000 P d84cd04486824a0c90066cd6e5708eb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d84cd04486824a0c90066cd6e5708eb7, State: Initialized, Role: FOLLOWER
I20260924 16:17:58.448253 20838 consensus_queue.cc:260] T 00000000000000000000000000000000 P d84cd04486824a0c90066cd6e5708eb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d84cd04486824a0c90066cd6e5708eb7" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 32969 } }
I20260924 16:17:58.448539 20838 raft_consensus.cc:388] T 00000000000000000000000000000000 P d84cd04486824a0c90066cd6e5708eb7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:17:58.448704 20838 raft_consensus.cc:482] T 00000000000000000000000000000000 P d84cd04486824a0c90066cd6e5708eb7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:17:58.448886 20838 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d84cd04486824a0c90066cd6e5708eb7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:17:58.451045 20838 raft_consensus.cc:504] T 00000000000000000000000000000000 P d84cd04486824a0c90066cd6e5708eb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d84cd04486824a0c90066cd6e5708eb7" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 32969 } }
I20260924 16:17:58.451510 20838 leader_election.cc:302] T 00000000000000000000000000000000 P d84cd04486824a0c90066cd6e5708eb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d84cd04486824a0c90066cd6e5708eb7; no voters:
I20260924 16:17:58.452016 20838 leader_election.cc:288] T 00000000000000000000000000000000 P d84cd04486824a0c90066cd6e5708eb7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:17:58.452162 20848 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d84cd04486824a0c90066cd6e5708eb7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:17:58.452883 20848 raft_consensus.cc:686] T 00000000000000000000000000000000 P d84cd04486824a0c90066cd6e5708eb7 [term 1 LEADER]: Becoming Leader. State: Replica: d84cd04486824a0c90066cd6e5708eb7, State: Running, Role: LEADER
I20260924 16:17:58.453486 20848 consensus_queue.cc:237] T 00000000000000000000000000000000 P d84cd04486824a0c90066cd6e5708eb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d84cd04486824a0c90066cd6e5708eb7" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 32969 } }
I20260924 16:17:58.454159 20838 sys_catalog.cc:564] T 00000000000000000000000000000000 P d84cd04486824a0c90066cd6e5708eb7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:17:58.456125 20849 sys_catalog.cc:455] T 00000000000000000000000000000000 P d84cd04486824a0c90066cd6e5708eb7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d84cd04486824a0c90066cd6e5708eb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d84cd04486824a0c90066cd6e5708eb7" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 32969 } } }
I20260924 16:17:58.456513 20849 sys_catalog.cc:458] T 00000000000000000000000000000000 P d84cd04486824a0c90066cd6e5708eb7 [sys.catalog]: This master's current role is: LEADER
I20260924 16:17:58.456959 20850 sys_catalog.cc:455] T 00000000000000000000000000000000 P d84cd04486824a0c90066cd6e5708eb7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d84cd04486824a0c90066cd6e5708eb7. Latest consensus state: current_term: 1 leader_uuid: "d84cd04486824a0c90066cd6e5708eb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d84cd04486824a0c90066cd6e5708eb7" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 32969 } } }
I20260924 16:17:58.457260 20850 sys_catalog.cc:458] T 00000000000000000000000000000000 P d84cd04486824a0c90066cd6e5708eb7 [sys.catalog]: This master's current role is: LEADER
I20260924 16:17:58.466076 20855 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:17:58.472855 20855 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:17:58.480991 20855 catalog_manager.cc:1347] Generated new cluster ID: 92983fecaaed434b82b66b207317ee59
I20260924 16:17:58.481118 20855 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:17:58.497665 20855 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:17:58.499544 20855 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:17:58.514022 20855 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d84cd04486824a0c90066cd6e5708eb7: Generated new TSK 0
I20260924 16:17:58.515055 20855 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:17:58.525918 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 20769
W20260924 16:17:58.527235 20832 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:55535
W20260924 16:17:58.528061 30632 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 20769: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 16:17:58.528230 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 20769
2026-09-24T16:17:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1967 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790266323169505-30632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:17:58 dist-test-slave-v5j9 krb5kdc[20871](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:17:58 dist-test-slave-v5j9 krb5kdc[20871](info): set up 2 sockets
Sep 24 16:17:58 dist-test-slave-v5j9 krb5kdc[20871](info): commencing operation
krb5kdc: starting...
W20260924 16:17:59.765686 30632 mini_kdc.cc:121] Time spent starting KDC: real 1.200s user 0.001s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Sep 24 16:17:59 dist-test-slave-v5j9 krb5kdc[20871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:17:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:17:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.234.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab.
Entry for principal kudu/127.29.234.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab.
I20260924 16:18:00.048787 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790266323169505-30632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790266323169505-30632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790266323169505-30632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:42525
--webserver_interface=127.29.234.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab
--principal=kudu/127.29.234.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:35609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.234.62:42525
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:18:00.391659 20884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:18:00.391970 20884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:18:00.392138 20884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:18:00.399472 20884 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:18:00.399600 20884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:18:00.399716 20884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:18:00.399819 20884 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:18:00.399895 20884 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:18:00.410053 20884 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:35609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790266323169505-30632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790266323169505-30632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.234.62:42525
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab
--principal=kudu/127.29.234.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:42525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.234.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790266323169505-30632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:18:00.411484 20884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:18:00.412886 20884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:18:00.424757 20889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:18:00.424831 20890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:18:00.424883 20892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:18:00.425922 20884 server_base.cc:1034] running on GCE node
I20260924 16:18:00.426791 20884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:18:00.428210 20884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:18:00.429446 20884 hybrid_clock.cc:648] HybridClock initialized: now 1790266680429390 us; error 44 us; skew 500 ppm
I20260924 16:18:00.429891 20884 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:18:00 dist-test-slave-v5j9 krb5kdc[20871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266680, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:18:00.433193 20884 init.cc:377] Logged in from keytab as kudu/127.29.234.62@KRBTEST.COM (short username kudu)
I20260924 16:18:00.434731 20884 webserver.cc:460] Webserver started at http://127.29.234.62:41163/ using document root <none> and password file <none>
I20260924 16:18:00.435407 20884 fs_manager.cc:359] Metadata directory not provided
I20260924 16:18:00.435511 20884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:18:00.435770 20884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:18:00.438921 20884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790266323169505-30632-0/minicluster-data/master-0/data/instance:
uuid: "d4ebc4aaeae44107a54719f81e786dbf"
format_stamp: "Formatted at 2026-09-24 16:18:00 on dist-test-slave-v5j9"
server_key: "7f0defdf0416f6552709b8f14af30892"
server_key_iv: "f4265bbf2a87074dc69314316b8481c7"
server_key_version: "encryptionkey@0"
I20260924 16:18:00.439653 20884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790266323169505-30632-0/minicluster-data/master-0/wal/instance:
uuid: "d4ebc4aaeae44107a54719f81e786dbf"
format_stamp: "Formatted at 2026-09-24 16:18:00 on dist-test-slave-v5j9"
server_key: "7f0defdf0416f6552709b8f14af30892"
server_key_iv: "f4265bbf2a87074dc69314316b8481c7"
server_key_version: "encryptionkey@0"
I20260924 16:18:00.445041 20884 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260924 16:18:00.448818 20899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:18:00.450435 20884 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:18:00.450599 20884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790266323169505-30632-0/minicluster-data/master-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790266323169505-30632-0/minicluster-data/master-0/wal
uuid: "d4ebc4aaeae44107a54719f81e786dbf"
format_stamp: "Formatted at 2026-09-24 16:18:00 on dist-test-slave-v5j9"
server_key: "7f0defdf0416f6552709b8f14af30892"
server_key_iv: "f4265bbf2a87074dc69314316b8481c7"
server_key_version: "encryptionkey@0"
I20260924 16:18:00.450783 20884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790266323169505-30632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790266323169505-30632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790266323169505-30632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:18:00.456897 20884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:18:00.461267 20884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:18:00.461632 20884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:18:00.480381 20884 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.62:42525
I20260924 16:18:00.480417 20950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.62:42525 every 8 connection(s)
I20260924 16:18:00.481897 20884 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
I20260924 16:18:00.485049 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 20884
I20260924 16:18:00.485270 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790266323169505-30632-0/minicluster-data/master-0/wal/instance
I20260924 16:18:00.485772 30632 external_mini_cluster.cc:1428] Setting key 5527c5f52e3cdc7f0d2392db60d922b8
I20260924 16:18:00.486042 20951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:18:00.493984 20954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:00.487584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37379 (local address 127.29.234.62:42525)
0924 16:18:00.488418 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0924 16:18:00.488429 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 16:18:00.488473 (+ 44us) server_negotiation.cc:407] Connection header received
0924 16:18:00.490616 (+ 2143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:18:00.490647 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:18:00.490977 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:18:00.491309 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:18:00.492362 (+ 1053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:18:00.492371 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:18:00.492383 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:18:00.492396 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:18:00.492460 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:18:00.492479 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:18:00.492661 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0924 16:18:00.492791 (+ 130us) server_negotiation.cc:299] Negotiation successful
0924 16:18:00.493002 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":360,"threads_started":1}
I20260924 16:18:00.497773 30632 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:18:00.499591 20951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4ebc4aaeae44107a54719f81e786dbf: Bootstrap starting.
I20260924 16:18:00.503266 20951 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4ebc4aaeae44107a54719f81e786dbf: Neither blocks nor log segments found. Creating new log.
I20260924 16:18:00.504521 20954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:00.501983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55513 (local address 127.29.234.62:42525)
0924 16:18:00.502219 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0924 16:18:00.502225 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:18:00.502643 (+ 418us) server_negotiation.cc:407] Connection header received
0924 16:18:00.502798 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:18:00.502802 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:18:00.502844 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:18:00.502918 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:18:00.503871 (+ 953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:18:00.503875 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:18:00.503878 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 16:18:00.503891 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:18:00.503926 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 16:18:00.503940 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:18:00.504023 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 16:18:00.504172 (+ 149us) server_negotiation.cc:299] Negotiation successful
0924 16:18:00.504218 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 16:18:00.504662 20951 log.cc:824] T 00000000000000000000000000000000 P d4ebc4aaeae44107a54719f81e786dbf: Log is configured to *not* fsync() on all Append() calls
I20260924 16:18:00.507470 20951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4ebc4aaeae44107a54719f81e786dbf: No bootstrap required, opened a new log
I20260924 16:18:00.515182 20951 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4ebc4aaeae44107a54719f81e786dbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4ebc4aaeae44107a54719f81e786dbf" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 42525 } }
I20260924 16:18:00.515679 20951 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4ebc4aaeae44107a54719f81e786dbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 16:18:00.515821 20951 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4ebc4aaeae44107a54719f81e786dbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4ebc4aaeae44107a54719f81e786dbf, State: Initialized, Role: FOLLOWER
I20260924 16:18:00.516487 20951 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4ebc4aaeae44107a54719f81e786dbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4ebc4aaeae44107a54719f81e786dbf" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 42525 } }
I20260924 16:18:00.516729 20951 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4ebc4aaeae44107a54719f81e786dbf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 16:18:00.516885 20951 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4ebc4aaeae44107a54719f81e786dbf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 16:18:00.517062 20951 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4ebc4aaeae44107a54719f81e786dbf [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:18:00.519335 20951 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4ebc4aaeae44107a54719f81e786dbf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4ebc4aaeae44107a54719f81e786dbf" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 42525 } }
I20260924 16:18:00.519894 20951 leader_election.cc:302] T 00000000000000000000000000000000 P d4ebc4aaeae44107a54719f81e786dbf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4ebc4aaeae44107a54719f81e786dbf; no voters:
I20260924 16:18:00.520505 20951 leader_election.cc:288] T 00000000000000000000000000000000 P d4ebc4aaeae44107a54719f81e786dbf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 16:18:00.520665 20961 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4ebc4aaeae44107a54719f81e786dbf [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:18:00.521982 20961 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4ebc4aaeae44107a54719f81e786dbf [term 1 LEADER]: Becoming Leader. State: Replica: d4ebc4aaeae44107a54719f81e786dbf, State: Running, Role: LEADER
I20260924 16:18:00.522637 20961 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4ebc4aaeae44107a54719f81e786dbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4ebc4aaeae44107a54719f81e786dbf" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 42525 } }
I20260924 16:18:00.523054 20951 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4ebc4aaeae44107a54719f81e786dbf [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:18:00.526105 20962 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4ebc4aaeae44107a54719f81e786dbf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4ebc4aaeae44107a54719f81e786dbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4ebc4aaeae44107a54719f81e786dbf" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 42525 } } }
I20260924 16:18:00.526084 20963 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4ebc4aaeae44107a54719f81e786dbf [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4ebc4aaeae44107a54719f81e786dbf. Latest consensus state: current_term: 1 leader_uuid: "d4ebc4aaeae44107a54719f81e786dbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4ebc4aaeae44107a54719f81e786dbf" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 42525 } } }
I20260924 16:18:00.526587 20962 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4ebc4aaeae44107a54719f81e786dbf [sys.catalog]: This master's current role is: LEADER
I20260924 16:18:00.526592 20963 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4ebc4aaeae44107a54719f81e786dbf [sys.catalog]: This master's current role is: LEADER
I20260924 16:18:00.528898 20970 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:18:00.533990 20970 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:18:00.543234 20970 catalog_manager.cc:1347] Generated new cluster ID: 45420feff1c54519b7f716f08459d396
I20260924 16:18:00.543375 20970 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:18:00.551417 20970 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:18:00.551954 20970 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:18:00.559794 20970 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4ebc4aaeae44107a54719f81e786dbf: Generated new TSK 0
I20260924 16:18:00.560539 20970 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:18:00.567091 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 20884
W20260924 16:18:00.568217 20946 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37379
W20260924 16:18:00.568871 30632 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 20884: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 16:18:00.569028 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 20884
2026-09-24T16:18:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2037 ms)
[----------] 18 tests from AuthTokenIssuingTest (35892 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266323169505-30632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:18:00 dist-test-slave-v5j9 krb5kdc[20983](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:18:00 dist-test-slave-v5j9 krb5kdc[20983](info): set up 2 sockets
Sep 24 16:18:00 dist-test-slave-v5j9 krb5kdc[20983](info): commencing operation
krb5kdc: starting...
W20260924 16:18:01.689651 30632 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Sep 24 16:18:01 dist-test-slave-v5j9 krb5kdc[20983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:18:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:18:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.234.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab.
Entry for principal kudu/127.29.234.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab.
I20260924 16:18:01.958752 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266323169505-30632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266323169505-30632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266323169505-30632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:34765
--webserver_interface=127.29.234.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab
--principal=kudu/127.29.234.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:38629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.234.62:34765,127.29.234.61:33311,127.29.234.60:38481
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:18:02.309010 20997 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 16:18:02.309501 20997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:18:02.309607 20997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:18:02.309767 20997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:18:02.317260 20997 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:18:02.317476 20997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:18:02.317592 20997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:18:02.317678 20997 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:18:02.317776 20997 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:18:02.328433 20997 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:38629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266323169505-30632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266323169505-30632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.234.62:34765,127.29.234.61:33311,127.29.234.60:38481
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab
--principal=kudu/127.29.234.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:34765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.234.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266323169505-30632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:18:02.329993 20997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:18:02.331367 20997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:18:02.343248 21004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:18:02.343310 21002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:18:02.343214 21006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:18:02.344103 20997 server_base.cc:1034] running on GCE node
I20260924 16:18:02.344945 20997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:18:02.346494 20997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:18:02.347731 20997 hybrid_clock.cc:648] HybridClock initialized: now 1790266682347658 us; error 71 us; skew 500 ppm
I20260924 16:18:02.348145 20997 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:18:02 dist-test-slave-v5j9 krb5kdc[20983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266682, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:18:02.351485 20997 init.cc:377] Logged in from keytab as kudu/127.29.234.62@KRBTEST.COM (short username kudu)
I20260924 16:18:02.352983 20997 webserver.cc:460] Webserver started at http://127.29.234.62:38373/ using document root <none> and password file <none>
I20260924 16:18:02.353776 20997 fs_manager.cc:359] Metadata directory not provided
I20260924 16:18:02.353868 20997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:18:02.354161 20997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:18:02.357177 20997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266323169505-30632-0/minicluster-data/master-0/data/instance:
uuid: "fe33eea882b5457f8d5c81c115e7fef5"
format_stamp: "Formatted at 2026-09-24 16:18:02 on dist-test-slave-v5j9"
server_key: "107368cb98a5c8ce31b898973d5e01f2"
server_key_iv: "dc41f5760fc114c6dac1c612f85ff969"
server_key_version: "encryptionkey@0"
I20260924 16:18:02.358044 20997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266323169505-30632-0/minicluster-data/master-0/wal/instance:
uuid: "fe33eea882b5457f8d5c81c115e7fef5"
format_stamp: "Formatted at 2026-09-24 16:18:02 on dist-test-slave-v5j9"
server_key: "107368cb98a5c8ce31b898973d5e01f2"
server_key_iv: "dc41f5760fc114c6dac1c612f85ff969"
server_key_version: "encryptionkey@0"
I20260924 16:18:02.363660 20997 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260924 16:18:02.367506 21013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:18:02.368988 20997 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 16:18:02.369243 20997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266323169505-30632-0/minicluster-data/master-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266323169505-30632-0/minicluster-data/master-0/wal
uuid: "fe33eea882b5457f8d5c81c115e7fef5"
format_stamp: "Formatted at 2026-09-24 16:18:02 on dist-test-slave-v5j9"
server_key: "107368cb98a5c8ce31b898973d5e01f2"
server_key_iv: "dc41f5760fc114c6dac1c612f85ff969"
server_key_version: "encryptionkey@0"
I20260924 16:18:02.369522 20997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266323169505-30632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266323169505-30632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266323169505-30632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:18:02.388653 20997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:18:02.392853 20997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:18:02.393149 20997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:18:02.412842 20997 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.62:34765
I20260924 16:18:02.412870 21066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.62:34765 every 8 connection(s)
I20260924 16:18:02.414331 20997 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
I20260924 16:18:02.415014 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 20997
I20260924 16:18:02.415292 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266323169505-30632-0/minicluster-data/master-0/wal/instance
I20260924 16:18:02.416042 30632 external_mini_cluster.cc:1428] Setting key 3a5942e1b28fe2e41b92b2bd17742bd8
I20260924 16:18:02.418889 21067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:18:02.425772 21067 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.234.62" port: 34765 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.29.234.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.61@KRBTEST.COM" created.
Sep 24 16:18:02 dist-test-slave-v5j9 krb5kdc[20983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266682, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.62@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:18:02.452517 21070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:02.429199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:34765 (local address 127.0.0.1:59992)
0924 16:18:02.430177 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0924 16:18:02.430235 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 16:18:02.432379 (+ 2144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:18:02.433260 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:18:02.433266 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:18:02.434055 (+ 789us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:18:02.434429 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:18:02.434450 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:18:02.436043 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:18:02.436049 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:18:02.437182 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:18:02.437194 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:18:02.438211 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:18:02.438219 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:18:02.438390 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:18:02.439083 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:18:02.439106 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:18:02.443289 (+ 4183us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:18:02.448015 (+ 4726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:18:02.448027 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:18:02.448039 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:18:02.448497 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:18:02.449007 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:18:02.449011 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:18:02.449013 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:18:02.449228 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:18:02.450367 (+ 1139us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:18:02.450374 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:18:02.450618 (+ 244us) client_negotiation.cc:769] Sending connection context
0924 16:18:02.450858 (+ 240us) client_negotiation.cc:240] Negotiation successful
0924 16:18:02.451322 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664,"thread_start_us":396,"threads_started":1}
I20260924 16:18:02.452589 21071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:02.429200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59992 (local address 127.29.234.62:34765)
0924 16:18:02.431145 (+ 1945us) server_negotiation.cc:206] Beginning negotiation
0924 16:18:02.431153 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:18:02.431217 (+ 64us) server_negotiation.cc:407] Connection header received
0924 16:18:02.432769 (+ 1552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:18:02.432775 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:18:02.433020 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:18:02.433253 (+ 233us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:18:02.434654 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:02.435884 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:02.437327 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:02.437963 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:02.438187 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:18:02.443567 (+ 5380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:18:02.443582 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:18:02.443605 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:18:02.443663 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:18:02.447811 (+ 4148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:18:02.448663 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:18:02.448676 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:18:02.448682 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:18:02.448887 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:18:02.449387 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:18:02.449395 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:18:02.449401 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:18:02.450204 (+ 803us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:18:02.450429 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0924 16:18:02.451069 (+ 640us) server_negotiation.cc:299] Negotiation successful
0924 16:18:02.451541 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":103,"server-negotiator.queue_time_us":1708,"spinlock_wait_cycles":19328,"thread_start_us":1468,"threads_started":1}
I20260924 16:18:02.455857 21067 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.234.61" port: 33311 } has no permanent_uuid. Determining permanent_uuid...
I20260924 16:18:02.457054 21070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:02.456502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.61:33311 (local address 127.0.0.1:46970)
0924 16:18:02.456739 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0924 16:18:02.456886 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.61:33311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260924 16:18:02.457422 21014 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.234.61:33311: connect: Connection refused (error 111)
I20260924 16:18:02.459239 21070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:02.458752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.61:33311 (local address 127.0.0.1:46986)
0924 16:18:02.458938 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0924 16:18:02.459044 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.61:33311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 16:18:02.459448 21067 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.234.61:33311: Network error: Client connection negotiation failed: client connection to 127.29.234.61:33311: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.61.keytab.
Entry for principal kudu/127.29.234.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.61.keytab.
I20260924 16:18:02.480902 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266323169505-30632-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266323169505-30632-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266323169505-30632-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266323169505-30632-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.61:33311
--webserver_interface=127.29.234.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.61.keytab
--principal=kudu/127.29.234.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:38629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.234.62:34765,127.29.234.61:33311,127.29.234.60:38481
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:18:02.508859 21067 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.234.61" port: 33311 } attempt: 1
I20260924 16:18:02.510265 21070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:02.509716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.61:33311 (local address 127.0.0.1:46994)
0924 16:18:02.509939 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0924 16:18:02.510095 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.61:33311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260924 16:18:02.511353 21070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:02.510891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.61:33311 (local address 127.0.0.1:47002)
0924 16:18:02.511079 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 16:18:02.511181 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.61:33311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260924 16:18:02.511547 21067 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.234.61:33311: Network error: Client connection negotiation failed: client connection to 127.29.234.61:33311: connect: Connection refused (error 111)
I20260924 16:18:02.579823 21067 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.234.61" port: 33311 } attempt: 2
I20260924 16:18:02.581184 21070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:02.580531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.61:33311 (local address 127.0.0.1:47016)
0924 16:18:02.580838 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0924 16:18:02.580978 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.61:33311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260924 16:18:02.582332 21070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:02.581932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.61:33311 (local address 127.0.0.1:47020)
0924 16:18:02.582082 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 16:18:02.582167 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.61:33311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 16:18:02.582522 21067 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.234.61:33311: Network error: Client connection negotiation failed: client connection to 127.29.234.61:33311: connect: Connection refused (error 111)
I20260924 16:18:02.673799 21067 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.234.61" port: 33311 } attempt: 3
I20260924 16:18:02.675060 21070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:02.674536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.61:33311 (local address 127.0.0.1:47028)
0924 16:18:02.674758 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0924 16:18:02.674874 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.61:33311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260924 16:18:02.676120 21070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:02.675719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.61:33311 (local address 127.0.0.1:47034)
0924 16:18:02.675866 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 16:18:02.675950 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.61:33311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260924 16:18:02.676306 21067 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.234.61:33311: Network error: Client connection negotiation failed: client connection to 127.29.234.61:33311: connect: Connection refused (error 111)
I20260924 16:18:02.819571 21067 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.234.61" port: 33311 } attempt: 4
I20260924 16:18:02.820930 21070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:02.820327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.61:33311 (local address 127.0.0.1:47036)
0924 16:18:02.820681 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0924 16:18:02.820787 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.61:33311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213}
I20260924 16:18:02.822073 21070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:02.821691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.61:33311 (local address 127.0.0.1:47040)
0924 16:18:02.821840 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 16:18:02.821937 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.61:33311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 16:18:02.822355 21067 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.234.61:33311: Network error: Client connection negotiation failed: client connection to 127.29.234.61:33311: connect: Connection refused (error 111)
W20260924 16:18:02.826170 21074 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 16:18:02.826568 21074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:18:02.826671 21074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:18:02.826808 21074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:18:02.834519 21074 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:18:02.834622 21074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:18:02.834708 21074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:18:02.834800 21074 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:18:02.834903 21074 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:18:02.845188 21074 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:38629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266323169505-30632-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266323169505-30632-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.29.234.62:34765,127.29.234.61:33311,127.29.234.60:38481
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.61.keytab
--principal=kudu/127.29.234.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.61:33311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266323169505-30632-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.29.234.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266323169505-30632-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:18:02.846642 21074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:18:02.848001 21074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:18:02.860082 21080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:18:02.860041 21082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:18:02.860162 21079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:18:02.860884 21074 server_base.cc:1034] running on GCE node
I20260924 16:18:02.861822 21074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:18:02.863390 21074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:18:02.864619 21074 hybrid_clock.cc:648] HybridClock initialized: now 1790266682864542 us; error 45 us; skew 500 ppm
I20260924 16:18:02.864992 21074 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:18:02 dist-test-slave-v5j9 krb5kdc[20983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266682, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:18:02.868409 21074 init.cc:377] Logged in from keytab as kudu/127.29.234.61@KRBTEST.COM (short username kudu)
I20260924 16:18:02.870067 21074 webserver.cc:460] Webserver started at http://127.29.234.61:38241/ using document root <none> and password file <none>
I20260924 16:18:02.870775 21074 fs_manager.cc:359] Metadata directory not provided
I20260924 16:18:02.870903 21074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:18:02.871227 21074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:18:02.874369 21074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266323169505-30632-0/minicluster-data/master-1/data/instance:
uuid: "f38b872fd23f448c992e056c224620de"
format_stamp: "Formatted at 2026-09-24 16:18:02 on dist-test-slave-v5j9"
server_key: "ada196352a56bfcda2d27b214e8bbe0c"
server_key_iv: "b67f4e471d4e27b3dce8ad8601cf430c"
server_key_version: "encryptionkey@0"
I20260924 16:18:02.875211 21074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266323169505-30632-0/minicluster-data/master-1/wal/instance:
uuid: "f38b872fd23f448c992e056c224620de"
format_stamp: "Formatted at 2026-09-24 16:18:02 on dist-test-slave-v5j9"
server_key: "ada196352a56bfcda2d27b214e8bbe0c"
server_key_iv: "b67f4e471d4e27b3dce8ad8601cf430c"
server_key_version: "encryptionkey@0"
I20260924 16:18:02.880600 21074 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 16:18:02.884179 21089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:18:02.885695 21074 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:18:02.885885 21074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266323169505-30632-0/minicluster-data/master-1/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266323169505-30632-0/minicluster-data/master-1/wal
uuid: "f38b872fd23f448c992e056c224620de"
format_stamp: "Formatted at 2026-09-24 16:18:02 on dist-test-slave-v5j9"
server_key: "ada196352a56bfcda2d27b214e8bbe0c"
server_key_iv: "b67f4e471d4e27b3dce8ad8601cf430c"
server_key_version: "encryptionkey@0"
I20260924 16:18:02.886124 21074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266323169505-30632-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266323169505-30632-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266323169505-30632-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:18:02.910132 21074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:18:02.914446 21074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:18:02.914685 21074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:18:02.933260 21074 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.61:33311
I20260924 16:18:02.933279 21142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.61:33311 every 8 connection(s)
I20260924 16:18:02.934736 21074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266323169505-30632-0/minicluster-data/master-1/data/info.pb
I20260924 16:18:02.937700 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 21074
I20260924 16:18:02.937893 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266323169505-30632-0/minicluster-data/master-1/wal/instance
I20260924 16:18:02.938354 30632 external_mini_cluster.cc:1428] Setting key 878bbc1f007c95e788f8510b64a19426
I20260924 16:18:02.938894 21144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:18:02.945806 21144 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.234.62" port: 34765 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 16:18:02 dist-test-slave-v5j9 krb5kdc[20983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266682, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.61@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.234.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.60@KRBTEST.COM" created.
I20260924 16:18:02.967797 21071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:02.948781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60008 (local address 127.29.234.62:34765)
0924 16:18:02.949017 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0924 16:18:02.949024 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:18:02.950331 (+ 1307us) server_negotiation.cc:407] Connection header received
0924 16:18:02.952289 (+ 1958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:18:02.952298 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:18:02.952395 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:18:02.952517 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:18:02.954155 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:02.955289 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:02.956826 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:02.957665 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:02.957807 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:18:02.961483 (+ 3676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:18:02.961493 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:18:02.961500 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:18:02.961543 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:18:02.964552 (+ 3009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:18:02.965400 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:18:02.965405 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:18:02.965409 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:18:02.965578 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:18:02.966007 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:18:02.966013 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:18:02.966016 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:18:02.966341 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:18:02.966496 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 16:18:02.967214 (+ 718us) server_negotiation.cc:299] Negotiation successful
0924 16:18:02.967456 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260924 16:18:02.968817 21147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:02.949235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:34765 (local address 127.0.0.1:60008)
0924 16:18:02.950136 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0924 16:18:02.950185 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 16:18:02.952003 (+ 1818us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:18:02.952754 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:18:02.952763 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:18:02.953306 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:18:02.953954 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:18:02.953972 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:18:02.955459 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:18:02.955463 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:18:02.956678 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:18:02.956690 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:18:02.957884 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:18:02.957888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:18:02.958080 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:18:02.958775 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:18:02.958805 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:18:02.961213 (+ 2408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:18:02.964744 (+ 3531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:18:02.964757 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:18:02.964777 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:18:02.965221 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:18:02.965690 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:18:02.965693 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:18:02.965695 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:18:02.965889 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:18:02.966478 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:18:02.966486 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:18:02.966848 (+ 362us) client_negotiation.cc:769] Sending connection context
0924 16:18:02.967124 (+ 276us) client_negotiation.cc:240] Negotiation successful
0924 16:18:02.967667 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":367,"threads_started":1}
I20260924 16:18:02.970605 21144 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.234.61" port: 33311 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 16:18:02 dist-test-slave-v5j9 krb5kdc[20983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266682, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.61@KRBTEST.COM for kudu/127.29.234.61@KRBTEST.COM
I20260924 16:18:02.987202 21147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:02.971269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.61:33311 (local address 127.0.0.1:47046)
0924 16:18:02.971517 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0924 16:18:02.971531 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 16:18:02.971613 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:18:02.972734 (+ 1121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:18:02.972742 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:18:02.973240 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:18:02.973497 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:18:02.973512 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:18:02.975229 (+ 1717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:18:02.975233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:18:02.976092 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:18:02.976102 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:18:02.977407 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:18:02.977412 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:18:02.977567 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:18:02.978154 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:18:02.978171 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:18:02.980401 (+ 2230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:18:02.984740 (+ 4339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:18:02.984744 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:18:02.984747 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:18:02.985127 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:18:02.985567 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:18:02.985571 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:18:02.985573 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:18:02.985707 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:18:02.986479 (+ 772us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:18:02.986483 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:18:02.986594 (+ 111us) client_negotiation.cc:769] Sending connection context
0924 16:18:02.986716 (+ 122us) client_negotiation.cc:240] Negotiation successful
0924 16:18:02.986921 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":1}
I20260924 16:18:02.987738 21148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:02.971587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47046 (local address 127.29.234.61:33311)
0924 16:18:02.972261 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0924 16:18:02.972270 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:18:02.972300 (+ 30us) server_negotiation.cc:407] Connection header received
0924 16:18:02.972350 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:18:02.972360 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:18:02.972570 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:18:02.972698 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:18:02.973667 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:02.975099 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:02.976247 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:02.977211 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:02.977383 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:18:02.980619 (+ 3236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:18:02.980634 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:18:02.980654 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:18:02.980695 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:18:02.984608 (+ 3913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:18:02.985275 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:18:02.985286 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:18:02.985292 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:18:02.985462 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:18:02.985824 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:18:02.985831 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:18:02.985837 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:18:02.986357 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:18:02.986559 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0924 16:18:02.986906 (+ 347us) server_negotiation.cc:299] Negotiation successful
0924 16:18:02.987208 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":498,"thread_start_us":340,"threads_started":1}
I20260924 16:18:02.990372 21144 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.234.60" port: 38481 } has no permanent_uuid. Determining permanent_uuid...
I20260924 16:18:02.991569 21147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:02.991084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.60:38481 (local address 127.0.0.1:37756)
0924 16:18:02.991282 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0924 16:18:02.991407 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.60:38481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260924 16:18:02.991840 21092 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.234.60:38481: connect: Connection refused (error 111)
I20260924 16:18:02.993515 21147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:02.993116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.60:38481 (local address 127.0.0.1:37758)
0924 16:18:02.993264 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 16:18:02.993406 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.60:38481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260924 16:18:02.993745 21144 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.234.60:38481: Network error: Client connection negotiation failed: client connection to 127.29.234.60:38481: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.60.keytab.
Entry for principal kudu/127.29.234.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.60.keytab.
I20260924 16:18:03.001030 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266323169505-30632-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266323169505-30632-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266323169505-30632-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266323169505-30632-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.60:38481
--webserver_interface=127.29.234.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.60.keytab
--principal=kudu/127.29.234.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:38629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.234.62:34765,127.29.234.61:33311,127.29.234.60:38481
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:18:03.043072 21144 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.234.60" port: 38481 } attempt: 1
I20260924 16:18:03.044360 21147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:03.043804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.60:38481 (local address 127.0.0.1:37764)
0924 16:18:03.044032 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0924 16:18:03.044174 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.60:38481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260924 16:18:03.045439 21147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:03.045023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.60:38481 (local address 127.0.0.1:37778)
0924 16:18:03.045198 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 16:18:03.045288 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.60:38481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260924 16:18:03.045653 21144 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.234.60:38481: Network error: Client connection negotiation failed: client connection to 127.29.234.60:38481: connect: Connection refused (error 111)
I20260924 16:18:03.113902 21144 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.234.60" port: 38481 } attempt: 2
I20260924 16:18:03.115237 21147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:03.114607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.60:38481 (local address 127.0.0.1:37794)
0924 16:18:03.114866 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0924 16:18:03.114999 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.60:38481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260924 16:18:03.116266 21147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:03.115882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.60:38481 (local address 127.0.0.1:37806)
0924 16:18:03.116035 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 16:18:03.116120 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.60:38481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 16:18:03.116500 21144 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.234.60:38481: Network error: Client connection negotiation failed: client connection to 127.29.234.60:38481: connect: Connection refused (error 111)
I20260924 16:18:03.121595 21067 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.234.61" port: 33311 } attempt: 5
Sep 24 16:18:03 dist-test-slave-v5j9 krb5kdc[20983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266682, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.62@KRBTEST.COM for kudu/127.29.234.61@KRBTEST.COM
I20260924 16:18:03.135771 21070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:03.122290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.61:33311 (local address 127.0.0.1:47052)
0924 16:18:03.122454 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 16:18:03.122485 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 16:18:03.122586 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:18:03.122946 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:18:03.122949 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:18:03.123367 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:18:03.123541 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:18:03.123550 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:18:03.124900 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:18:03.124904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:18:03.125832 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:18:03.125845 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:18:03.126795 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:18:03.126799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:18:03.126972 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:18:03.127577 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:18:03.127595 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:18:03.129763 (+ 2168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:18:03.133323 (+ 3560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:18:03.133326 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:18:03.133329 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:18:03.133739 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:18:03.134191 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:18:03.134194 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:18:03.134196 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:18:03.134392 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:18:03.135063 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:18:03.135066 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:18:03.135229 (+ 163us) client_negotiation.cc:769] Sending connection context
0924 16:18:03.135335 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 16:18:03.135532 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260924 16:18:03.136180 21148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:03.122321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47052 (local address 127.29.234.61:33311)
0924 16:18:03.122505 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 16:18:03.122511 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:18:03.122534 (+ 23us) server_negotiation.cc:407] Connection header received
0924 16:18:03.122703 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:18:03.122708 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:18:03.122785 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:18:03.122923 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:18:03.123684 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:03.124769 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:03.125968 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:03.126653 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:03.126788 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:18:03.129936 (+ 3148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:18:03.129942 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:18:03.129946 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:18:03.129981 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:18:03.133201 (+ 3220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:18:03.133857 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:18:03.133865 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:18:03.133870 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:18:03.134055 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:18:03.134520 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:18:03.134527 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:18:03.134532 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:18:03.134944 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:18:03.135119 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0924 16:18:03.135523 (+ 404us) server_negotiation.cc:299] Negotiation successful
0924 16:18:03.135787 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 16:18:03.137140 21067 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.234.60" port: 38481 } has no permanent_uuid. Determining permanent_uuid...
I20260924 16:18:03.138322 21070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:03.137931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.60:38481 (local address 127.0.0.1:37814)
0924 16:18:03.138093 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 16:18:03.138197 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.60:38481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260924 16:18:03.139278 21070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:03.138978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.60:38481 (local address 127.0.0.1:37824)
0924 16:18:03.139114 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 16:18:03.139178 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.60:38481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260924 16:18:03.139464 21067 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.234.60:38481: Network error: Client connection negotiation failed: client connection to 127.29.234.60:38481: connect: Connection refused (error 111)
I20260924 16:18:03.190685 21067 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.234.60" port: 38481 } attempt: 1
I20260924 16:18:03.191895 21070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:03.191422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.60:38481 (local address 127.0.0.1:37836)
0924 16:18:03.191622 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0924 16:18:03.191725 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.60:38481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260924 16:18:03.192919 21070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:03.192582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.60:38481 (local address 127.0.0.1:37844)
0924 16:18:03.192746 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 16:18:03.192815 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.60:38481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260924 16:18:03.193133 21067 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.234.60:38481: Network error: Client connection negotiation failed: client connection to 127.29.234.60:38481: connect: Connection refused (error 111)
I20260924 16:18:03.207744 21144 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.234.60" port: 38481 } attempt: 3
I20260924 16:18:03.209121 21147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:03.208489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.60:38481 (local address 127.0.0.1:37846)
0924 16:18:03.208744 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0924 16:18:03.208883 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.60:38481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260924 16:18:03.210446 21147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:03.209874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.60:38481 (local address 127.0.0.1:37848)
0924 16:18:03.210177 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0924 16:18:03.210281 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.60:38481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
W20260924 16:18:03.210731 21144 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.234.60:38481: Network error: Client connection negotiation failed: client connection to 127.29.234.60:38481: connect: Connection refused (error 111)
I20260924 16:18:03.261384 21067 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.234.60" port: 38481 } attempt: 2
I20260924 16:18:03.262575 21070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:03.262094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.60:38481 (local address 127.0.0.1:37856)
0924 16:18:03.262302 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0924 16:18:03.262415 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.60:38481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260924 16:18:03.263943 21070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:03.263470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.60:38481 (local address 127.0.0.1:37868)
0924 16:18:03.263674 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0924 16:18:03.263787 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.60:38481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260924 16:18:03.264222 21067 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.234.60:38481: Network error: Client connection negotiation failed: client connection to 127.29.234.60:38481: connect: Connection refused (error 111)
W20260924 16:18:03.348021 21151 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 16:18:03.348356 21151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:18:03.348428 21151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:18:03.348511 21151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 16:18:03.353998 21144 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.234.60" port: 38481 } attempt: 4
I20260924 16:18:03.355248 21147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:03.354698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.60:38481 (local address 127.0.0.1:37878)
0924 16:18:03.354946 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0924 16:18:03.355052 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.60:38481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260924 16:18:03.356218 21147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:03.355877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.60:38481 (local address 127.0.0.1:37884)
0924 16:18:03.356016 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 16:18:03.356091 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.60:38481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260924 16:18:03.356319 21151 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:18:03.356428 21144 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.234.60:38481: Network error: Client connection negotiation failed: client connection to 127.29.234.60:38481: connect: Connection refused (error 111)
W20260924 16:18:03.356468 21151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:18:03.356565 21151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:18:03.356622 21151 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:18:03.356689 21151 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:18:03.366809 21151 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:38629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266323169505-30632-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266323169505-30632-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.29.234.62:34765,127.29.234.61:33311,127.29.234.60:38481
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.60.keytab
--principal=kudu/127.29.234.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.60:38481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266323169505-30632-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.29.234.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266323169505-30632-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:18:03.368160 21151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:18:03.369634 21151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 16:18:03.370450 21067 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.234.60" port: 38481 } attempt: 3
I20260924 16:18:03.371685 21070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:03.371122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.60:38481 (local address 127.0.0.1:37886)
0924 16:18:03.371382 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0924 16:18:03.371490 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.60:38481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260924 16:18:03.372784 21070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:03.372325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.60:38481 (local address 127.0.0.1:37890)
0924 16:18:03.372529 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0924 16:18:03.372648 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.60:38481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260924 16:18:03.372987 21067 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.234.60:38481: Network error: Client connection negotiation failed: client connection to 127.29.234.60:38481: connect: Connection refused (error 111)
W20260924 16:18:03.382798 21157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:18:03.382767 21159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:18:03.383994 21156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:18:03.385035 21151 server_base.cc:1034] running on GCE node
I20260924 16:18:03.385986 21151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:18:03.387352 21151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:18:03.388561 21151 hybrid_clock.cc:648] HybridClock initialized: now 1790266683388476 us; error 59 us; skew 500 ppm
I20260924 16:18:03.388860 21151 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:18:03 dist-test-slave-v5j9 krb5kdc[20983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266683, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:18:03.401141 21151 init.cc:377] Logged in from keytab as kudu/127.29.234.60@KRBTEST.COM (short username kudu)
I20260924 16:18:03.402935 21151 webserver.cc:460] Webserver started at http://127.29.234.60:38709/ using document root <none> and password file <none>
I20260924 16:18:03.403676 21151 fs_manager.cc:359] Metadata directory not provided
I20260924 16:18:03.403802 21151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:18:03.404124 21151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:18:03.407315 21151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266323169505-30632-0/minicluster-data/master-2/data/instance:
uuid: "44ee165799b24bb5980e3f2d5cdb2a66"
format_stamp: "Formatted at 2026-09-24 16:18:03 on dist-test-slave-v5j9"
server_key: "1abe3d0e1140b1e0ccb877ec0de604e9"
server_key_iv: "ec24208f4f2263c7ba8925ca3ddf61e1"
server_key_version: "encryptionkey@0"
I20260924 16:18:03.408124 21151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266323169505-30632-0/minicluster-data/master-2/wal/instance:
uuid: "44ee165799b24bb5980e3f2d5cdb2a66"
format_stamp: "Formatted at 2026-09-24 16:18:03 on dist-test-slave-v5j9"
server_key: "1abe3d0e1140b1e0ccb877ec0de604e9"
server_key_iv: "ec24208f4f2263c7ba8925ca3ddf61e1"
server_key_version: "encryptionkey@0"
I20260924 16:18:03.413791 21151 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260924 16:18:03.417516 21166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:18:03.418924 21151 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:18:03.419106 21151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266323169505-30632-0/minicluster-data/master-2/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266323169505-30632-0/minicluster-data/master-2/wal
uuid: "44ee165799b24bb5980e3f2d5cdb2a66"
format_stamp: "Formatted at 2026-09-24 16:18:03 on dist-test-slave-v5j9"
server_key: "1abe3d0e1140b1e0ccb877ec0de604e9"
server_key_iv: "ec24208f4f2263c7ba8925ca3ddf61e1"
server_key_version: "encryptionkey@0"
I20260924 16:18:03.419345 21151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266323169505-30632-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266323169505-30632-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266323169505-30632-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:18:03.431442 21151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:18:03.436197 21151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:18:03.436486 21151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:18:03.456024 21151 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.60:38481
I20260924 16:18:03.456010 21218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.60:38481 every 8 connection(s)
I20260924 16:18:03.457571 21151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266323169505-30632-0/minicluster-data/master-2/data/info.pb
I20260924 16:18:03.460129 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 21151
I20260924 16:18:03.460321 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790266323169505-30632-0/minicluster-data/master-2/wal/instance
I20260924 16:18:03.460883 30632 external_mini_cluster.cc:1428] Setting key 309417243b6a9bcae6925dc627cc2ec3
I20260924 16:18:03.462380 21219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:18:03.472047 21219 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.234.62" port: 34765 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 16:18:03 dist-test-slave-v5j9 krb5kdc[20983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.234.60@KRBTEST.COM
Sep 24 16:18:03 dist-test-slave-v5j9 krb5kdc[20983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266683, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.60@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:18:03.486598 21222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:03.462983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37902 (local address 127.29.234.60:38481)
0924 16:18:03.463812 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0924 16:18:03.463825 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 16:18:03.463872 (+ 47us) server_negotiation.cc:407] Connection header received
0924 16:18:03.465505 (+ 1633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:18:03.465532 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:18:03.466021 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:18:03.466431 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:18:03.468394 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:03.470003 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:03.471742 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:03.472504 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:03.472661 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:18:03.477685 (+ 5024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:18:03.477697 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:18:03.477711 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:18:03.477759 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:18:03.481671 (+ 3912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:18:03.482587 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:18:03.482596 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:18:03.482605 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:18:03.482815 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:18:03.483325 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:18:03.483344 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:18:03.483348 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:18:03.484139 (+ 791us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:18:03.484444 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0924 16:18:03.484889 (+ 445us) server_negotiation.cc:299] Negotiation successful
0924 16:18:03.485424 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":384,"threads_started":1}
I20260924 16:18:03.492641 21224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:03.475458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:34765 (local address 127.0.0.1:60022)
0924 16:18:03.476421 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0924 16:18:03.476475 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 16:18:03.476671 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:18:03.477104 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:18:03.477110 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:18:03.478022 (+ 912us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:18:03.478401 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:18:03.478416 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:18:03.479704 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:18:03.479710 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:18:03.481548 (+ 1838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:18:03.481565 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:18:03.482617 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:18:03.482623 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:18:03.482815 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:18:03.483494 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:18:03.483516 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:18:03.486156 (+ 2640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:18:03.489424 (+ 3268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:18:03.489439 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:18:03.489445 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:18:03.490207 (+ 762us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:18:03.490651 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:18:03.490657 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:18:03.490660 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:18:03.490893 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:18:03.491438 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:18:03.491450 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:18:03.491742 (+ 292us) client_negotiation.cc:769] Sending connection context
0924 16:18:03.491966 (+ 224us) client_negotiation.cc:240] Negotiation successful
0924 16:18:03.492307 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":717,"thread_start_us":559,"threads_started":1}
I20260924 16:18:03.492741 21223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:03.475459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60022 (local address 127.29.234.62:34765)
0924 16:18:03.476031 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0924 16:18:03.476038 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:18:03.476575 (+ 537us) server_negotiation.cc:407] Connection header received
0924 16:18:03.476812 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:18:03.476820 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:18:03.476917 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:18:03.477038 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:18:03.478562 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:03.479478 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:03.481712 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:03.482387 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:03.482542 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:18:03.486354 (+ 3812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:18:03.486358 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:18:03.486362 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:18:03.486397 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:18:03.489247 (+ 2850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:18:03.490337 (+ 1090us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:18:03.490342 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:18:03.490345 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:18:03.490503 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:18:03.491012 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:18:03.491016 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:18:03.491018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:18:03.491271 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:18:03.491381 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 16:18:03.492057 (+ 676us) server_negotiation.cc:299] Negotiation successful
0924 16:18:03.492247 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":297,"threads_started":1}
I20260924 16:18:03.494059 21219 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.234.61" port: 33311 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 16:18:03 dist-test-slave-v5j9 krb5kdc[20983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
Sep 24 16:18:03 dist-test-slave-v5j9 krb5kdc[20983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266683, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.60@KRBTEST.COM for kudu/127.29.234.61@KRBTEST.COM
I20260924 16:18:03.507225 21225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:03.489650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60026 (local address 127.29.234.62:34765)
0924 16:18:03.490291 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0924 16:18:03.490299 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:18:03.490342 (+ 43us) server_negotiation.cc:407] Connection header received
0924 16:18:03.490386 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:18:03.490392 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:18:03.490468 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:18:03.490579 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:18:03.492326 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:03.493569 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:03.495444 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:03.496112 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:03.496248 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:18:03.500820 (+ 4572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:18:03.500828 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:18:03.500832 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:18:03.500873 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:18:03.504081 (+ 3208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:18:03.505041 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:18:03.505045 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:18:03.505047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:18:03.505184 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:18:03.505673 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:18:03.505677 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:18:03.505679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:18:03.505968 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:18:03.506301 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0924 16:18:03.506593 (+ 292us) server_negotiation.cc:299] Negotiation successful
0924 16:18:03.506833 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":278,"threads_started":1}
I20260924 16:18:03.508894 21224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:03.494732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.61:33311 (local address 127.0.0.1:47060)
0924 16:18:03.494893 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 16:18:03.494912 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 16:18:03.495005 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:18:03.495374 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:18:03.495394 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:18:03.495865 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:18:03.496043 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:18:03.496055 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:18:03.497498 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:18:03.497502 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:18:03.498399 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:18:03.498411 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:18:03.499396 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:18:03.499403 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:18:03.499590 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:18:03.500158 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:18:03.500175 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:18:03.502358 (+ 2183us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:18:03.506090 (+ 3732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:18:03.506096 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:18:03.506101 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:18:03.506523 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:18:03.506952 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:18:03.506958 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:18:03.506962 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:18:03.507184 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:18:03.507826 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:18:03.507831 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:18:03.508015 (+ 184us) client_negotiation.cc:769] Sending connection context
0924 16:18:03.508185 (+ 170us) client_negotiation.cc:240] Negotiation successful
0924 16:18:03.508505 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260924 16:18:03.508971 21148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:03.494853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47060 (local address 127.29.234.61:33311)
0924 16:18:03.495047 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 16:18:03.495054 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:18:03.495076 (+ 22us) server_negotiation.cc:407] Connection header received
0924 16:18:03.495142 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:18:03.495147 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:18:03.495245 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:18:03.495366 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:18:03.496285 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:03.497334 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:03.498546 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:03.499242 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:03.499380 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:18:03.502552 (+ 3172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:18:03.502556 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:18:03.502559 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:18:03.502593 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:18:03.505915 (+ 3322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:18:03.506668 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:18:03.506674 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:18:03.506678 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:18:03.506826 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:18:03.507312 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:18:03.507317 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:18:03.507321 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:18:03.507696 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:18:03.507867 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0924 16:18:03.508287 (+ 420us) server_negotiation.cc:299] Negotiation successful
0924 16:18:03.508535 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 16:18:03.510406 21219 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.234.60" port: 38481 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 16:18:03 dist-test-slave-v5j9 krb5kdc[20983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266683, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.60@KRBTEST.COM for kudu/127.29.234.60@KRBTEST.COM
Sep 24 16:18:03 dist-test-slave-v5j9 krb5kdc[20983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.234.61@KRBTEST.COM
I20260924 16:18:03.525414 21224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:03.511390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.60:38481 (local address 127.0.0.1:37912)
0924 16:18:03.511711 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0924 16:18:03.511735 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 16:18:03.511856 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:18:03.512158 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:18:03.512165 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:18:03.512676 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:18:03.512852 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:18:03.512863 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:18:03.513895 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:18:03.513903 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:18:03.514763 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:18:03.514773 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:18:03.515462 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:18:03.515467 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:18:03.515643 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:18:03.516149 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:18:03.516164 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:18:03.518517 (+ 2353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:18:03.522713 (+ 4196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:18:03.522717 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:18:03.522720 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:18:03.523115 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:18:03.523548 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:18:03.523552 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:18:03.523555 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:18:03.523721 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:18:03.524374 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:18:03.524381 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:18:03.524540 (+ 159us) client_negotiation.cc:769] Sending connection context
0924 16:18:03.524660 (+ 120us) client_negotiation.cc:240] Negotiation successful
0924 16:18:03.524990 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215}
I20260924 16:18:03.525765 21222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:03.511374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37912 (local address 127.29.234.60:38481)
0924 16:18:03.511540 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 16:18:03.511545 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:18:03.511808 (+ 263us) server_negotiation.cc:407] Connection header received
0924 16:18:03.511954 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:18:03.511958 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:18:03.512012 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:18:03.512082 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:18:03.512987 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:03.513780 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:03.514896 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:03.515311 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:03.515443 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:18:03.518717 (+ 3274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:18:03.518725 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:18:03.518730 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:18:03.518779 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:18:03.522538 (+ 3759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:18:03.523236 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:18:03.523244 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:18:03.523249 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:18:03.523415 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:18:03.523855 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:18:03.523860 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:18:03.523864 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:18:03.524241 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:18:03.524366 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 16:18:03.524803 (+ 437us) server_negotiation.cc:299] Negotiation successful
0924 16:18:03.525227 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":114,"server-negotiator.queue_time_us":40}
I20260924 16:18:03.526997 21148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:03.508952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47064 (local address 127.29.234.61:33311)
0924 16:18:03.509233 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0924 16:18:03.509239 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:18:03.509258 (+ 19us) server_negotiation.cc:407] Connection header received
0924 16:18:03.509507 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:18:03.509515 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:18:03.509595 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:18:03.509685 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:18:03.511658 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:03.512945 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:03.514320 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:03.514771 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:03.514922 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:18:03.520117 (+ 5195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:18:03.520124 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:18:03.520129 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:18:03.520178 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:18:03.523737 (+ 3559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:18:03.524477 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:18:03.524481 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:18:03.524484 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:18:03.524661 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:18:03.525103 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:18:03.525112 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:18:03.525116 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:18:03.525524 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:18:03.525854 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0924 16:18:03.526372 (+ 518us) server_negotiation.cc:299] Negotiation successful
0924 16:18:03.526645 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":433,"threads_started":1}
I20260924 16:18:03.531739 21219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44ee165799b24bb5980e3f2d5cdb2a66: Bootstrap starting.
I20260924 16:18:03.532047 30632 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:18:03.535590 21219 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44ee165799b24bb5980e3f2d5cdb2a66: Neither blocks nor log segments found. Creating new log.
I20260924 16:18:03.536991 21219 log.cc:824] T 00000000000000000000000000000000 P 44ee165799b24bb5980e3f2d5cdb2a66: Log is configured to *not* fsync() on all Append() calls
I20260924 16:18:03.534789 21041 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "fe33eea882b5457f8d5c81c115e7fef5"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60026
I20260924 16:18:03.540427 21219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44ee165799b24bb5980e3f2d5cdb2a66: No bootstrap required, opened a new log
I20260924 16:18:03.550240 21067 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.234.60" port: 38481 } attempt: 4
I20260924 16:18:03.551937 21219 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44ee165799b24bb5980e3f2d5cdb2a66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33eea882b5457f8d5c81c115e7fef5" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 34765 } } peers { permanent_uuid: "f38b872fd23f448c992e056c224620de" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 33311 } } peers { permanent_uuid: "44ee165799b24bb5980e3f2d5cdb2a66" member_type: VOTER last_known_addr { host: "127.29.234.60" port: 38481 } }
I20260924 16:18:03.552421 21219 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44ee165799b24bb5980e3f2d5cdb2a66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44ee165799b24bb5980e3f2d5cdb2a66, State: Initialized, Role: FOLLOWER
I20260924 16:18:03.552997 21219 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44ee165799b24bb5980e3f2d5cdb2a66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33eea882b5457f8d5c81c115e7fef5" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 34765 } } peers { permanent_uuid: "f38b872fd23f448c992e056c224620de" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 33311 } } peers { permanent_uuid: "44ee165799b24bb5980e3f2d5cdb2a66" member_type: VOTER last_known_addr { host: "127.29.234.60" port: 38481 } }
I20260924 16:18:03.554096 21228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44ee165799b24bb5980e3f2d5cdb2a66 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33eea882b5457f8d5c81c115e7fef5" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 34765 } } peers { permanent_uuid: "f38b872fd23f448c992e056c224620de" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 33311 } } peers { permanent_uuid: "44ee165799b24bb5980e3f2d5cdb2a66" member_type: VOTER last_known_addr { host: "127.29.234.60" port: 38481 } } }
I20260924 16:18:03.554447 21228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44ee165799b24bb5980e3f2d5cdb2a66 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:18:03.555449 21219 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44ee165799b24bb5980e3f2d5cdb2a66 [sys.catalog]: configured and running, proceeding with master startup.
Sep 24 16:18:03 dist-test-slave-v5j9 krb5kdc[20983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266682, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.62@KRBTEST.COM for kudu/127.29.234.60@KRBTEST.COM
W20260924 16:18:03.568434 21239 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 44ee165799b24bb5980e3f2d5cdb2a66: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260924 16:18:03.568540 21070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:03.551052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.60:38481 (local address 127.0.0.1:37928)
0924 16:18:03.551255 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0924 16:18:03.551279 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 16:18:03.551428 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:18:03.551787 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:18:03.551792 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:18:03.552305 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:18:03.552536 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:18:03.552552 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:18:03.554045 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:18:03.554050 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:18:03.555434 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:18:03.555453 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:18:03.556418 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:18:03.556425 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:18:03.556642 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:18:03.557520 (+ 878us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:18:03.557589 (+ 69us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:18:03.560591 (+ 3002us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:18:03.565087 (+ 4496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:18:03.565092 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:18:03.565096 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:18:03.565659 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:18:03.566283 (+ 624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:18:03.566288 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:18:03.566291 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:18:03.566473 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:18:03.567441 (+ 968us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:18:03.567446 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:18:03.567629 (+ 183us) client_negotiation.cc:769] Sending connection context
0924 16:18:03.567788 (+ 159us) client_negotiation.cc:240] Negotiation successful
0924 16:18:03.568122 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
W20260924 16:18:03.568707 21239 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 16:18:03.570128 21222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:03.551114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37928 (local address 127.29.234.60:38481)
0924 16:18:03.551348 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0924 16:18:03.551352 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:18:03.551372 (+ 20us) server_negotiation.cc:407] Connection header received
0924 16:18:03.551573 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:18:03.551577 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:18:03.551654 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:18:03.551728 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:18:03.552712 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:03.553901 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:03.555604 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:03.556247 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:03.556366 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:18:03.560954 (+ 4588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:18:03.560960 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:18:03.560968 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:18:03.561009 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:18:03.564951 (+ 3942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:18:03.565855 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:18:03.565860 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:18:03.565865 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:18:03.566086 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:18:03.566652 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:18:03.566657 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:18:03.566661 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:18:03.567139 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:18:03.567338 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0924 16:18:03.568946 (+ 1608us) server_negotiation.cc:299] Negotiation successful
0924 16:18:03.569402 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 16:18:03.565147 21151 master_runner.cc:186] Error getting master registration for 127.29.234.62:34765: OK, instance_id { permanent_uuid: "fe33eea882b5457f8d5c81c115e7fef5" instance_seqno: 1790266682407347 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 16:18:03.575707 21151 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 16:18:03.576330 21067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe33eea882b5457f8d5c81c115e7fef5: Bootstrap starting.
I20260924 16:18:03.580682 21067 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe33eea882b5457f8d5c81c115e7fef5: Neither blocks nor log segments found. Creating new log.
I20260924 16:18:03.582154 21067 log.cc:824] T 00000000000000000000000000000000 P fe33eea882b5457f8d5c81c115e7fef5: Log is configured to *not* fsync() on all Append() calls
I20260924 16:18:03.586031 21067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe33eea882b5457f8d5c81c115e7fef5: No bootstrap required, opened a new log
I20260924 16:18:03.595000 21067 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe33eea882b5457f8d5c81c115e7fef5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33eea882b5457f8d5c81c115e7fef5" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 34765 } } peers { permanent_uuid: "f38b872fd23f448c992e056c224620de" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 33311 } } peers { permanent_uuid: "44ee165799b24bb5980e3f2d5cdb2a66" member_type: VOTER last_known_addr { host: "127.29.234.60" port: 38481 } }
I20260924 16:18:03.595551 21067 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe33eea882b5457f8d5c81c115e7fef5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe33eea882b5457f8d5c81c115e7fef5, State: Initialized, Role: FOLLOWER
I20260924 16:18:03.596196 21067 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe33eea882b5457f8d5c81c115e7fef5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33eea882b5457f8d5c81c115e7fef5" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 34765 } } peers { permanent_uuid: "f38b872fd23f448c992e056c224620de" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 33311 } } peers { permanent_uuid: "44ee165799b24bb5980e3f2d5cdb2a66" member_type: VOTER last_known_addr { host: "127.29.234.60" port: 38481 } }
I20260924 16:18:03.597283 21242 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe33eea882b5457f8d5c81c115e7fef5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33eea882b5457f8d5c81c115e7fef5" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 34765 } } peers { permanent_uuid: "f38b872fd23f448c992e056c224620de" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 33311 } } peers { permanent_uuid: "44ee165799b24bb5980e3f2d5cdb2a66" member_type: VOTER last_known_addr { host: "127.29.234.60" port: 38481 } } }
I20260924 16:18:03.597743 21242 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe33eea882b5457f8d5c81c115e7fef5 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:18:03.598935 21067 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe33eea882b5457f8d5c81c115e7fef5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:18:03.599004 21041 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe33eea882b5457f8d5c81c115e7fef5 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 16:18:03.599243 21041 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe33eea882b5457f8d5c81c115e7fef5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:18:03.602212 21041 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe33eea882b5457f8d5c81c115e7fef5 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33eea882b5457f8d5c81c115e7fef5" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 34765 } } peers { permanent_uuid: "f38b872fd23f448c992e056c224620de" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 33311 } } peers { permanent_uuid: "44ee165799b24bb5980e3f2d5cdb2a66" member_type: VOTER last_known_addr { host: "127.29.234.60" port: 38481 } }
I20260924 16:18:03.604519 21193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fe33eea882b5457f8d5c81c115e7fef5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "44ee165799b24bb5980e3f2d5cdb2a66"
I20260924 16:18:03.605074 21193 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44ee165799b24bb5980e3f2d5cdb2a66 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:18:03.608657 21193 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 44ee165799b24bb5980e3f2d5cdb2a66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe33eea882b5457f8d5c81c115e7fef5 in term 1.
I20260924 16:18:03.609061 21041 leader_election.cc:288] T 00000000000000000000000000000000 P fe33eea882b5457f8d5c81c115e7fef5 [CANDIDATE]: Term 1 election: Requested vote from peers f38b872fd23f448c992e056c224620de (127.29.234.61:33311), 44ee165799b24bb5980e3f2d5cdb2a66 (127.29.234.60:38481)
I20260924 16:18:03.609416 21016 leader_election.cc:302] T 00000000000000000000000000000000 P fe33eea882b5457f8d5c81c115e7fef5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44ee165799b24bb5980e3f2d5cdb2a66, fe33eea882b5457f8d5c81c115e7fef5; no voters:
I20260924 16:18:03.609938 21242 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe33eea882b5457f8d5c81c115e7fef5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:18:03.626595 21242 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe33eea882b5457f8d5c81c115e7fef5 [term 1 LEADER]: Becoming Leader. State: Replica: fe33eea882b5457f8d5c81c115e7fef5, State: Running, Role: LEADER
I20260924 16:18:03.628286 21242 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe33eea882b5457f8d5c81c115e7fef5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33eea882b5457f8d5c81c115e7fef5" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 34765 } } peers { permanent_uuid: "f38b872fd23f448c992e056c224620de" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 33311 } } peers { permanent_uuid: "44ee165799b24bb5980e3f2d5cdb2a66" member_type: VOTER last_known_addr { host: "127.29.234.60" port: 38481 } }
W20260924 16:18:03.646420 21019 tablet.cc:2369] T 00000000000000000000000000000000 P fe33eea882b5457f8d5c81c115e7fef5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:18:03.646030 21247 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe33eea882b5457f8d5c81c115e7fef5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe33eea882b5457f8d5c81c115e7fef5. Latest consensus state: current_term: 1 leader_uuid: "fe33eea882b5457f8d5c81c115e7fef5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33eea882b5457f8d5c81c115e7fef5" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 34765 } } peers { permanent_uuid: "f38b872fd23f448c992e056c224620de" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 33311 } } peers { permanent_uuid: "44ee165799b24bb5980e3f2d5cdb2a66" member_type: VOTER last_known_addr { host: "127.29.234.60" port: 38481 } } }
I20260924 16:18:03.647235 21247 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe33eea882b5457f8d5c81c115e7fef5 [sys.catalog]: This master's current role is: LEADER
I20260924 16:18:03.656033 21144 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.234.60" port: 38481 } attempt: 5
I20260924 16:18:03.667346 21262 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:18:03.668515 21266 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
Sep 24 16:18:03 dist-test-slave-v5j9 krb5kdc[20983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266682, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.61@KRBTEST.COM for kudu/127.29.234.60@KRBTEST.COM
I20260924 16:18:03.685369 21225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:03.631980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60038 (local address 127.29.234.62:34765)
0924 16:18:03.632282 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0924 16:18:03.632288 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:18:03.642270 (+ 9982us) server_negotiation.cc:407] Connection header received
0924 16:18:03.645597 (+ 3327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:18:03.645608 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:18:03.645741 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:18:03.645851 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:18:03.653527 (+ 7676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:03.655188 (+ 1661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:03.669488 (+ 14300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:03.670308 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:03.670425 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:18:03.676700 (+ 6275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:18:03.676708 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:18:03.676712 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:18:03.676760 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:18:03.680966 (+ 4206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:18:03.682064 (+ 1098us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:18:03.682072 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:18:03.682093 (+ 21us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:18:03.682277 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:18:03.682763 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:18:03.682768 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:18:03.682773 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:18:03.683185 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:18:03.683498 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0924 16:18:03.684578 (+ 1080us) server_negotiation.cc:299] Negotiation successful
0924 16:18:03.684855 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157}
I20260924 16:18:03.685416 21262 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:18:03.674351 20997 master_runner.cc:186] Error getting master registration for 127.29.234.61:33311: OK, instance_id { permanent_uuid: "f38b872fd23f448c992e056c224620de" instance_seqno: 1790266682928022 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 16:18:03.688158 20997 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260924 16:18:03.688771 21172 tablet.cc:2369] T 00000000000000000000000000000000 P 44ee165799b24bb5980e3f2d5cdb2a66: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:18:03.690861 21222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:03.626713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37940 (local address 127.29.234.60:38481)
0924 16:18:03.626921 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0924 16:18:03.626927 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:18:03.638935 (+ 12008us) server_negotiation.cc:407] Connection header received
0924 16:18:03.640437 (+ 1502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:18:03.640447 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:18:03.640553 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:18:03.640640 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:18:03.647563 (+ 6923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:03.648871 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:03.654685 (+ 5814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:03.655663 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:03.655796 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:18:03.672480 (+ 16684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:18:03.672488 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:18:03.672494 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:18:03.672538 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:18:03.676920 (+ 4382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:18:03.680883 (+ 3963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:18:03.680889 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:18:03.680893 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:18:03.681063 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:18:03.681514 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:18:03.681519 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:18:03.681523 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:18:03.681920 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:18:03.682210 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0924 16:18:03.690214 (+ 8004us) server_negotiation.cc:299] Negotiation successful
0924 16:18:03.690445 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 16:18:03.694566 21148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:03.639275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47072 (local address 127.29.234.61:33311)
0924 16:18:03.639461 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 16:18:03.639466 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:18:03.647422 (+ 7956us) server_negotiation.cc:407] Connection header received
0924 16:18:03.651715 (+ 4293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:18:03.651724 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:18:03.651831 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:18:03.651919 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:18:03.658716 (+ 6797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:03.660175 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:03.667302 (+ 7127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:03.668030 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:03.668171 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:18:03.676805 (+ 8634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:18:03.676813 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:18:03.676817 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:18:03.676856 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:18:03.680431 (+ 3575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:18:03.691414 (+ 10983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:18:03.691422 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:18:03.691425 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:18:03.691720 (+ 295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:18:03.692228 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:18:03.692233 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:18:03.692236 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:18:03.692609 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:18:03.692912 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0924 16:18:03.693651 (+ 739us) server_negotiation.cc:299] Negotiation successful
0924 16:18:03.693986 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":71,"spinlock_wait_cycles":21888}
I20260924 16:18:03.695039 21147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:03.656773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.60:38481 (local address 127.0.0.1:37946)
0924 16:18:03.657256 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0924 16:18:03.657278 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 16:18:03.657422 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:18:03.661917 (+ 4495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:18:03.661924 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:18:03.662504 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:18:03.662746 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:18:03.662760 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:18:03.667411 (+ 4651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:18:03.667419 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:18:03.669136 (+ 1717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:18:03.669156 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:18:03.670628 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:18:03.670636 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:18:03.670877 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:18:03.671697 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:18:03.671720 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:18:03.675324 (+ 3604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:18:03.691412 (+ 16088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:18:03.691418 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:18:03.691424 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:18:03.692043 (+ 619us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:18:03.692812 (+ 769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:18:03.692818 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:18:03.692823 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:18:03.693008 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:18:03.694040 (+ 1032us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:18:03.694046 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:18:03.694204 (+ 158us) client_negotiation.cc:769] Sending connection context
0924 16:18:03.694331 (+ 127us) client_negotiation.cc:240] Negotiation successful
0924 16:18:03.694634 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302}
I20260924 16:18:03.696856 21264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:03.657676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37946 (local address 127.29.234.60:38481)
0924 16:18:03.660629 (+ 2953us) server_negotiation.cc:206] Beginning negotiation
0924 16:18:03.660635 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:18:03.660661 (+ 26us) server_negotiation.cc:407] Connection header received
0924 16:18:03.660718 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:18:03.660725 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:18:03.660827 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:18:03.662953 (+ 2126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:18:03.663297 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:03.664753 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:03.669519 (+ 4766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:03.670455 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:03.670619 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:18:03.681522 (+ 10903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:18:03.681530 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:18:03.681535 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:18:03.681592 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:18:03.691157 (+ 9565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:18:03.692349 (+ 1192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:18:03.692356 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:18:03.692360 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:18:03.692625 (+ 265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:18:03.693202 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:18:03.693208 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:18:03.693211 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:18:03.693838 (+ 627us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:18:03.695337 (+ 1499us) server_negotiation.cc:1035] Waiting for connection context
0924 16:18:03.695547 (+ 210us) server_negotiation.cc:299] Negotiation successful
0924 16:18:03.695886 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2788,"thread_start_us":2507,"threads_started":1}
I20260924 16:18:03.704939 21144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f38b872fd23f448c992e056c224620de: Bootstrap starting.
I20260924 16:18:03.706483 21193 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 44ee165799b24bb5980e3f2d5cdb2a66 [term 1 FOLLOWER]: Refusing update from remote peer fe33eea882b5457f8d5c81c115e7fef5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:18:03.707643 21247 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fe33eea882b5457f8d5c81c115e7fef5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44ee165799b24bb5980e3f2d5cdb2a66" member_type: VOTER last_known_addr { host: "127.29.234.60" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:18:03.711340 21144 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f38b872fd23f448c992e056c224620de: Neither blocks nor log segments found. Creating new log.
I20260924 16:18:03.713514 21144 log.cc:824] T 00000000000000000000000000000000 P f38b872fd23f448c992e056c224620de: Log is configured to *not* fsync() on all Append() calls
I20260924 16:18:03.730093 21144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f38b872fd23f448c992e056c224620de: No bootstrap required, opened a new log
I20260924 16:18:03.735862 21228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44ee165799b24bb5980e3f2d5cdb2a66 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe33eea882b5457f8d5c81c115e7fef5. Latest consensus state: current_term: 1 leader_uuid: "fe33eea882b5457f8d5c81c115e7fef5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33eea882b5457f8d5c81c115e7fef5" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 34765 } } peers { permanent_uuid: "f38b872fd23f448c992e056c224620de" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 33311 } } peers { permanent_uuid: "44ee165799b24bb5980e3f2d5cdb2a66" member_type: VOTER last_known_addr { host: "127.29.234.60" port: 38481 } } }
I20260924 16:18:03.736239 21228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44ee165799b24bb5980e3f2d5cdb2a66 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:18:03.749087 21144 raft_consensus.cc:348] T 00000000000000000000000000000000 P f38b872fd23f448c992e056c224620de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33eea882b5457f8d5c81c115e7fef5" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 34765 } } peers { permanent_uuid: "f38b872fd23f448c992e056c224620de" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 33311 } } peers { permanent_uuid: "44ee165799b24bb5980e3f2d5cdb2a66" member_type: VOTER last_known_addr { host: "127.29.234.60" port: 38481 } }
I20260924 16:18:03.750072 21144 raft_consensus.cc:729] T 00000000000000000000000000000000 P f38b872fd23f448c992e056c224620de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f38b872fd23f448c992e056c224620de, State: Initialized, Role: FOLLOWER
I20260924 16:18:03.750758 21144 consensus_queue.cc:260] T 00000000000000000000000000000000 P f38b872fd23f448c992e056c224620de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33eea882b5457f8d5c81c115e7fef5" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 34765 } } peers { permanent_uuid: "f38b872fd23f448c992e056c224620de" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 33311 } } peers { permanent_uuid: "44ee165799b24bb5980e3f2d5cdb2a66" member_type: VOTER last_known_addr { host: "127.29.234.60" port: 38481 } }
I20260924 16:18:03.751765 21268 mvcc.cc:205] Tried to move back new op lower bound from 7332932336437649408 to 7332932336169213952. Current Snapshot: MvccSnapshot[applied={T|T < 7332932336437649408}]
I20260924 16:18:03.756419 21269 mvcc.cc:205] Tried to move back new op lower bound from 7332932336437649408 to 7332932336169213952. Current Snapshot: MvccSnapshot[applied={T|T < 7332932336437649408}]
I20260924 16:18:03.764703 21247 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe33eea882b5457f8d5c81c115e7fef5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fe33eea882b5457f8d5c81c115e7fef5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33eea882b5457f8d5c81c115e7fef5" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 34765 } } peers { permanent_uuid: "f38b872fd23f448c992e056c224620de" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 33311 } } peers { permanent_uuid: "44ee165799b24bb5980e3f2d5cdb2a66" member_type: VOTER last_known_addr { host: "127.29.234.60" port: 38481 } } }
I20260924 16:18:03.765327 21247 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe33eea882b5457f8d5c81c115e7fef5 [sys.catalog]: This master's current role is: LEADER
I20260924 16:18:03.770447 21228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44ee165799b24bb5980e3f2d5cdb2a66 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fe33eea882b5457f8d5c81c115e7fef5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33eea882b5457f8d5c81c115e7fef5" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 34765 } } peers { permanent_uuid: "f38b872fd23f448c992e056c224620de" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 33311 } } peers { permanent_uuid: "44ee165799b24bb5980e3f2d5cdb2a66" member_type: VOTER last_known_addr { host: "127.29.234.60" port: 38481 } } }
I20260924 16:18:03.770764 21228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44ee165799b24bb5980e3f2d5cdb2a66 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:18:03.775477 21262 catalog_manager.cc:1347] Generated new cluster ID: 54c3dd16e56a45e8bb8dafbd32de2539
I20260924 16:18:03.775597 21262 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:18:03.799144 21272 sys_catalog.cc:455] T 00000000000000000000000000000000 P f38b872fd23f448c992e056c224620de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33eea882b5457f8d5c81c115e7fef5" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 34765 } } peers { permanent_uuid: "f38b872fd23f448c992e056c224620de" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 33311 } } peers { permanent_uuid: "44ee165799b24bb5980e3f2d5cdb2a66" member_type: VOTER last_known_addr { host: "127.29.234.60" port: 38481 } } }
I20260924 16:18:03.800851 21272 sys_catalog.cc:458] T 00000000000000000000000000000000 P f38b872fd23f448c992e056c224620de [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:18:03.802469 21116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fe33eea882b5457f8d5c81c115e7fef5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f38b872fd23f448c992e056c224620de"
I20260924 16:18:03.802908 21116 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f38b872fd23f448c992e056c224620de [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:18:03.806128 21116 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f38b872fd23f448c992e056c224620de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe33eea882b5457f8d5c81c115e7fef5 in term 1.
I20260924 16:18:03.807142 21115 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f38b872fd23f448c992e056c224620de [term 1 FOLLOWER]: Refusing update from remote peer fe33eea882b5457f8d5c81c115e7fef5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:18:03.808597 21247 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fe33eea882b5457f8d5c81c115e7fef5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f38b872fd23f448c992e056c224620de" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 33311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:18:03.814555 21144 sys_catalog.cc:564] T 00000000000000000000000000000000 P f38b872fd23f448c992e056c224620de [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:18:03.850334 21272 sys_catalog.cc:455] T 00000000000000000000000000000000 P f38b872fd23f448c992e056c224620de [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe33eea882b5457f8d5c81c115e7fef5. Latest consensus state: current_term: 1 leader_uuid: "fe33eea882b5457f8d5c81c115e7fef5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33eea882b5457f8d5c81c115e7fef5" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 34765 } } peers { permanent_uuid: "f38b872fd23f448c992e056c224620de" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 33311 } } peers { permanent_uuid: "44ee165799b24bb5980e3f2d5cdb2a66" member_type: VOTER last_known_addr { host: "127.29.234.60" port: 38481 } } }
I20260924 16:18:03.850754 21272 sys_catalog.cc:458] T 00000000000000000000000000000000 P f38b872fd23f448c992e056c224620de [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:18:03.850342 21279 sys_catalog.cc:455] T 00000000000000000000000000000000 P f38b872fd23f448c992e056c224620de [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fe33eea882b5457f8d5c81c115e7fef5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33eea882b5457f8d5c81c115e7fef5" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 34765 } } peers { permanent_uuid: "f38b872fd23f448c992e056c224620de" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 33311 } } peers { permanent_uuid: "44ee165799b24bb5980e3f2d5cdb2a66" member_type: VOTER last_known_addr { host: "127.29.234.60" port: 38481 } } }
I20260924 16:18:03.850957 21279 sys_catalog.cc:458] T 00000000000000000000000000000000 P f38b872fd23f448c992e056c224620de [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:18:03.860893 21247 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe33eea882b5457f8d5c81c115e7fef5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fe33eea882b5457f8d5c81c115e7fef5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe33eea882b5457f8d5c81c115e7fef5" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 34765 } } peers { permanent_uuid: "f38b872fd23f448c992e056c224620de" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 33311 } } peers { permanent_uuid: "44ee165799b24bb5980e3f2d5cdb2a66" member_type: VOTER last_known_addr { host: "127.29.234.60" port: 38481 } } }
I20260924 16:18:03.861248 21247 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe33eea882b5457f8d5c81c115e7fef5 [sys.catalog]: This master's current role is: LEADER
I20260924 16:18:03.870589 21262 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:18:03.873525 21262 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:18:03.908407 21262 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe33eea882b5457f8d5c81c115e7fef5: Generated new TSK 0
I20260924 16:18:03.910120 21262 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:18:03.932865 21289 catalog_manager.cc:1259] Loaded cluster ID: 54c3dd16e56a45e8bb8dafbd32de2539
I20260924 16:18:03.932988 21289 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f38b872fd23f448c992e056c224620de: loading cluster ID for follower catalog manager: success
I20260924 16:18:03.936729 21289 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f38b872fd23f448c992e056c224620de: acquiring CA information for follower catalog manager: success
I20260924 16:18:03.939054 21289 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f38b872fd23f448c992e056c224620de: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 16:18:04.045140 21225 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:18:04.030320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60046 (local address 127.29.234.62:34765)
0924 16:18:04.030536 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0924 16:18:04.030546 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:18:04.043025 (+ 12479us) server_negotiation.cc:407] Connection header received
0924 16:18:04.043355 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:18:04.043363 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:18:04.043478 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:18:04.043558 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:18:04.044841 (+ 1283us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60046: BlockingRecv error: recv got EOF from 127.0.0.1:60046 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260924 16:18:04.048950 21264 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:18:04.037328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37952 (local address 127.29.234.60:38481)
0924 16:18:04.037600 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0924 16:18:04.037610 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:18:04.046828 (+ 9218us) server_negotiation.cc:407] Connection header received
0924 16:18:04.047001 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:18:04.047009 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:18:04.047122 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:18:04.047236 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:18:04.048632 (+ 1396us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37952: BlockingRecv error: recv got EOF from 127.0.0.1:37952 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260924 16:18:04.052949 21148 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:18:04.042052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47084 (local address 127.29.234.61:33311)
0924 16:18:04.042357 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0924 16:18:04.042365 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:18:04.050740 (+ 8375us) server_negotiation.cc:407] Connection header received
0924 16:18:04.051591 (+ 851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:18:04.051598 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:18:04.051693 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:18:04.051770 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:18:04.052710 (+ 940us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47084: BlockingRecv error: recv got EOF from 127.0.0.1:47084 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
I20260924 16:18:04.109110 21148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:04.090390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47096 (local address 127.29.234.61:33311)
0924 16:18:04.090605 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0924 16:18:04.090611 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:18:04.100406 (+ 9795us) server_negotiation.cc:407] Connection header received
0924 16:18:04.101063 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:18:04.101072 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:18:04.101372 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:18:04.101545 (+ 173us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:18:04.102385 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:04.103588 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:04.105741 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:04.106464 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:04.106573 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:18:04.107415 (+ 842us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:18:04.108189 (+ 774us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:18:04.108343 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 16:18:04.108623 (+ 280us) server_negotiation.cc:299] Negotiation successful
0924 16:18:04.108735 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
W20260924 16:18:04.158413 21264 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:18:04.147773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37958 (local address 127.29.234.60:38481)
0924 16:18:04.148082 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0924 16:18:04.148090 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:18:04.156097 (+ 8007us) server_negotiation.cc:407] Connection header received
0924 16:18:04.156565 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:18:04.156572 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:18:04.156726 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:18:04.156821 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:18:04.158097 (+ 1276us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37958: BlockingRecv error: recv got EOF from 127.0.0.1:37958 (error 108)
Metrics: {"server-negotiator.queue_time_us":168}
I20260924 16:18:04.200959 21148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:04.193948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47102 (local address 127.29.234.61:33311)
0924 16:18:04.194146 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 16:18:04.194153 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:18:04.194174 (+ 21us) server_negotiation.cc:407] Connection header received
0924 16:18:04.194471 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:18:04.194478 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:18:04.194573 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:18:04.194638 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:18:04.195358 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:04.196579 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:04.198605 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:04.199274 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:04.199393 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:18:04.200046 (+ 653us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:18:04.200220 (+ 174us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:18:04.200302 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 16:18:04.200565 (+ 263us) server_negotiation.cc:299] Negotiation successful
0924 16:18:04.200660 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260924 16:18:04.244308 21264 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:18:04.234660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37960 (local address 127.29.234.60:38481)
0924 16:18:04.234857 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 16:18:04.234864 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:18:04.242240 (+ 7376us) server_negotiation.cc:407] Connection header received
0924 16:18:04.242741 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:18:04.242748 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:18:04.242848 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:18:04.242924 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:18:04.244041 (+ 1117us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37960: BlockingRecv error: recv got EOF from 127.0.0.1:37960 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 16:18:04.297917 21148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:04.279739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47110 (local address 127.29.234.61:33311)
0924 16:18:04.279918 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 16:18:04.279924 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:18:04.290865 (+ 10941us) server_negotiation.cc:407] Connection header received
0924 16:18:04.291219 (+ 354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:18:04.291227 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:18:04.291346 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:18:04.291411 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:18:04.292320 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:04.293585 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:04.295610 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:04.296275 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:04.296387 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:18:04.296999 (+ 612us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:18:04.297150 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:18:04.297210 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 16:18:04.297482 (+ 272us) server_negotiation.cc:299] Negotiation successful
0924 16:18:04.297577 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260924 16:18:04.308826 21264 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:18:04.306834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37966 (local address 127.29.234.60:38481)
0924 16:18:04.307001 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 16:18:04.307005 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:18:04.307449 (+ 444us) server_negotiation.cc:407] Connection header received
0924 16:18:04.307596 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:18:04.307603 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:18:04.307706 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:18:04.307830 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:18:04.308550 (+ 720us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37966: BlockingRecv error: recv got EOF from 127.0.0.1:37966 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 16:18:04.320858 21148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:04.316010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47122 (local address 127.29.234.61:33311)
0924 16:18:04.316269 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0924 16:18:04.316276 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:18:04.316301 (+ 25us) server_negotiation.cc:407] Connection header received
0924 16:18:04.316406 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:18:04.316411 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:18:04.316489 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:18:04.316581 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:18:04.316995 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:04.317832 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:04.319151 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:04.319639 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:04.319772 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:18:04.320165 (+ 393us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:18:04.320269 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:18:04.320365 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 16:18:04.320502 (+ 137us) server_negotiation.cc:299] Negotiation successful
0924 16:18:04.320581 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
W20260924 16:18:04.329448 21264 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:18:04.327543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37972 (local address 127.29.234.60:38481)
0924 16:18:04.327810 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0924 16:18:04.327817 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:18:04.328032 (+ 215us) server_negotiation.cc:407] Connection header received
0924 16:18:04.328168 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:18:04.328174 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:18:04.328274 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:18:04.328389 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:18:04.329100 (+ 711us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37972: BlockingRecv error: recv got EOF from 127.0.0.1:37972 (error 108)
Metrics: {"server-negotiator.queue_time_us":124}
I20260924 16:18:04.349135 21148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:04.343235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47138 (local address 127.29.234.61:33311)
0924 16:18:04.343454 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 16:18:04.343460 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:18:04.343781 (+ 321us) server_negotiation.cc:407] Connection header received
0924 16:18:04.343944 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:18:04.343950 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:18:04.344048 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:18:04.344164 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:18:04.344527 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:04.345844 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:04.347177 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:04.347869 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:04.348020 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:18:04.348355 (+ 335us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:18:04.348505 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:18:04.348605 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 16:18:04.348700 (+ 95us) server_negotiation.cc:299] Negotiation successful
0924 16:18:04.348800 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
W20260924 16:18:04.357873 21264 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:18:04.355535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37984 (local address 127.29.234.60:38481)
0924 16:18:04.355772 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0924 16:18:04.355789 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 16:18:04.356223 (+ 434us) server_negotiation.cc:407] Connection header received
0924 16:18:04.356446 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:18:04.356450 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:18:04.356641 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:18:04.356741 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:18:04.357630 (+ 889us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37984: BlockingRecv error: recv got EOF from 127.0.0.1:37984 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 16:18:04.384049 21148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:04.377446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47144 (local address 127.29.234.61:33311)
0924 16:18:04.377682 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0924 16:18:04.377688 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:18:04.378033 (+ 345us) server_negotiation.cc:407] Connection header received
0924 16:18:04.378339 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:18:04.378346 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:18:04.378438 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:18:04.378507 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:18:04.379249 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:04.380103 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:04.382164 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:04.382592 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:04.382709 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:18:04.383262 (+ 553us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:18:04.383380 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:18:04.383446 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 16:18:04.383662 (+ 216us) server_negotiation.cc:299] Negotiation successful
0924 16:18:04.383735 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
W20260924 16:18:04.392186 21264 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:18:04.390032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37996 (local address 127.29.234.60:38481)
0924 16:18:04.390222 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 16:18:04.390230 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:18:04.390648 (+ 418us) server_negotiation.cc:407] Connection header received
0924 16:18:04.390866 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:18:04.390874 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:18:04.390964 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:18:04.391071 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:18:04.391942 (+ 871us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37996: BlockingRecv error: recv got EOF from 127.0.0.1:37996 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 16:18:04.439000 21148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:04.432808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47152 (local address 127.29.234.61:33311)
0924 16:18:04.433042 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0924 16:18:04.433051 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:18:04.433487 (+ 436us) server_negotiation.cc:407] Connection header received
0924 16:18:04.433712 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:18:04.433717 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:18:04.433815 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:18:04.433911 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:18:04.434327 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:04.435546 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:04.436856 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:04.437594 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:04.437710 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:18:04.438139 (+ 429us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:18:04.438301 (+ 162us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:18:04.438417 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 16:18:04.438546 (+ 129us) server_negotiation.cc:299] Negotiation successful
0924 16:18:04.438651 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260924 16:18:04.447511 21264 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:18:04.445303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38012 (local address 127.29.234.60:38481)
0924 16:18:04.445601 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0924 16:18:04.445611 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:18:04.446156 (+ 545us) server_negotiation.cc:407] Connection header received
0924 16:18:04.446354 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:18:04.446359 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:18:04.446430 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:18:04.446531 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:18:04.447276 (+ 745us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38012: BlockingRecv error: recv got EOF from 127.0.0.1:38012 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
I20260924 16:18:04.520732 21148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:04.515533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47154 (local address 127.29.234.61:33311)
0924 16:18:04.515733 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 16:18:04.515738 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:18:04.516251 (+ 513us) server_negotiation.cc:407] Connection header received
0924 16:18:04.516392 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:18:04.516396 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:18:04.516469 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:18:04.516559 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:18:04.516997 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:04.517849 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:04.519236 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:04.519652 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:04.519755 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:18:04.520146 (+ 391us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:18:04.520249 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:18:04.520326 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 16:18:04.520460 (+ 134us) server_negotiation.cc:299] Negotiation successful
0924 16:18:04.520524 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260924 16:18:04.528676 21264 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:18:04.526824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38028 (local address 127.29.234.60:38481)
0924 16:18:04.527062 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0924 16:18:04.527068 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:18:04.527339 (+ 271us) server_negotiation.cc:407] Connection header received
0924 16:18:04.527507 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:18:04.527511 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:18:04.527581 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:18:04.527677 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:18:04.528425 (+ 748us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38028: BlockingRecv error: recv got EOF from 127.0.0.1:38028 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 16:18:04.571043 21239 catalog_manager.cc:1259] Loaded cluster ID: 54c3dd16e56a45e8bb8dafbd32de2539
I20260924 16:18:04.571162 21239 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 44ee165799b24bb5980e3f2d5cdb2a66: loading cluster ID for follower catalog manager: success
I20260924 16:18:04.574154 21239 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 44ee165799b24bb5980e3f2d5cdb2a66: acquiring CA information for follower catalog manager: success
I20260924 16:18:04.576037 21239 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 44ee165799b24bb5980e3f2d5cdb2a66: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 16:18:04.665513 21148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:04.660174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47158 (local address 127.29.234.61:33311)
0924 16:18:04.660333 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 16:18:04.660337 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:18:04.660571 (+ 234us) server_negotiation.cc:407] Connection header received
0924 16:18:04.660742 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:18:04.660746 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:18:04.660812 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:18:04.660899 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:18:04.661473 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:04.662290 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:04.663776 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:04.664263 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:04.664365 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:18:04.664799 (+ 434us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:18:04.664913 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:18:04.664980 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 16:18:04.665139 (+ 159us) server_negotiation.cc:299] Negotiation successful
0924 16:18:04.665204 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 16:18:04.677218 21264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:04.671245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38038 (local address 127.29.234.60:38481)
0924 16:18:04.671568 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0924 16:18:04.671598 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0924 16:18:04.671860 (+ 262us) server_negotiation.cc:407] Connection header received
0924 16:18:04.672071 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:18:04.672077 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:18:04.672331 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:18:04.672451 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:18:04.672812 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:04.673657 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:04.674929 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:04.675357 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:04.675491 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:18:04.676002 (+ 511us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:18:04.676615 (+ 613us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:18:04.676772 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 16:18:04.676829 (+ 57us) server_negotiation.cc:299] Negotiation successful
0924 16:18:04.676912 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169}
I20260924 16:18:04.692636 21397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:04.683303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60056 (local address 127.29.234.62:34765)
0924 16:18:04.684129 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0924 16:18:04.684138 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:18:04.684857 (+ 719us) server_negotiation.cc:407] Connection header received
0924 16:18:04.685014 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:18:04.685024 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:18:04.685293 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:18:04.685541 (+ 248us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:18:04.686265 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:04.687818 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:04.690401 (+ 2583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:04.690901 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:04.691050 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:18:04.691604 (+ 554us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:18:04.691977 (+ 373us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:18:04.692090 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 16:18:04.692223 (+ 133us) server_negotiation.cc:299] Negotiation successful
0924 16:18:04.692306 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":678,"thread_start_us":546,"threads_started":1}
I20260924 16:18:04.696308 21264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:04.683505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38048 (local address 127.29.234.60:38481)
0924 16:18:04.683720 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0924 16:18:04.683729 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:18:04.690173 (+ 6444us) server_negotiation.cc:407] Connection header received
0924 16:18:04.690360 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:18:04.690365 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:18:04.690444 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:18:04.690528 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:18:04.691087 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:04.692164 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:04.694195 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:04.694850 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:04.694973 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:18:04.695458 (+ 485us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:18:04.695593 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:18:04.695688 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 16:18:04.695905 (+ 217us) server_negotiation.cc:299] Negotiation successful
0924 16:18:04.695999 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 16:18:04.697103 21148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:04.684751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47172 (local address 127.29.234.61:33311)
0924 16:18:04.685005 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0924 16:18:04.685014 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:18:04.690350 (+ 5336us) server_negotiation.cc:407] Connection header received
0924 16:18:04.690716 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:18:04.690723 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:18:04.690825 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:18:04.690896 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:18:04.691987 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:04.693035 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:04.694961 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:04.695571 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:04.695654 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:18:04.696329 (+ 675us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:18:04.696463 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:18:04.696537 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 16:18:04.696771 (+ 234us) server_negotiation.cc:299] Negotiation successful
0924 16:18:04.696854 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 16:18:04.698033 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 20997
I20260924 16:18:04.699142 21062 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:18:04.811380 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 20997
I20260924 16:18:04.833200 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 21074
I20260924 16:18:04.834489 21138 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:18:04.942782 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 21074
I20260924 16:18:04.964507 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 21151
I20260924 16:18:04.965749 21214 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:18:05.074134 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 21151
2026-09-24T16:18:05Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4512 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266323169505-30632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 16:18:05 dist-test-slave-v5j9 krb5kdc[21406](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 16:18:05 dist-test-slave-v5j9 krb5kdc[21406](info): set up 2 sockets
Sep 24 16:18:05 dist-test-slave-v5j9 krb5kdc[21406](info): commencing operation
krb5kdc: starting...
W20260924 16:18:06.205286 30632 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.003s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266323169505-30632-0/krb5kdc/test-user.keytab.
Sep 24 16:18:06 dist-test-slave-v5j9 krb5kdc[21406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266686, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T16:18:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T16:18:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.234.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab.
Entry for principal kudu/127.29.234.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab.
I20260924 16:18:06.480264 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266323169505-30632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266323169505-30632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266323169505-30632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:37915
--webserver_interface=127.29.234.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab
--principal=kudu/127.29.234.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:35043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.234.62:37915,127.29.234.61:34023,127.29.234.60:41183
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 16:18:06.823470 21419 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 16:18:06.823849 21419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:18:06.823968 21419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:18:06.824097 21419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:18:06.831398 21419 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:18:06.831491 21419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:18:06.831595 21419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:18:06.831692 21419 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:18:06.831799 21419 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:18:06.845737 21419 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:35043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266323169505-30632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266323169505-30632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.234.62:37915,127.29.234.61:34023,127.29.234.60:41183
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.62.keytab
--principal=kudu/127.29.234.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.62:37915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.234.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266323169505-30632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:18:06.848079 21419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:18:06.849720 21419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:18:06.867060 21424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:18:06.866964 21427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:18:06.867009 21425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:18:06.867969 21419 server_base.cc:1034] running on GCE node
I20260924 16:18:06.868819 21419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:18:06.870414 21419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:18:06.871681 21419 hybrid_clock.cc:648] HybridClock initialized: now 1790266686871594 us; error 33 us; skew 500 ppm
I20260924 16:18:06.872165 21419 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:18:06 dist-test-slave-v5j9 krb5kdc[21406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266686, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:18:06.875627 21419 init.cc:377] Logged in from keytab as kudu/127.29.234.62@KRBTEST.COM (short username kudu)
I20260924 16:18:06.877175 21419 webserver.cc:460] Webserver started at http://127.29.234.62:34757/ using document root <none> and password file <none>
I20260924 16:18:06.877987 21419 fs_manager.cc:359] Metadata directory not provided
I20260924 16:18:06.878118 21419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:18:06.878477 21419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:18:06.881742 21419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266323169505-30632-0/minicluster-data/master-0/data/instance:
uuid: "0559e481e0c544cfb875ac477e5a1d0a"
format_stamp: "Formatted at 2026-09-24 16:18:06 on dist-test-slave-v5j9"
server_key: "1de1e11b43c4dfc2c1d6f7b3c649930a"
server_key_iv: "cb9abe8524d103f74b16da7c37d607b4"
server_key_version: "encryptionkey@0"
I20260924 16:18:06.882616 21419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266323169505-30632-0/minicluster-data/master-0/wal/instance:
uuid: "0559e481e0c544cfb875ac477e5a1d0a"
format_stamp: "Formatted at 2026-09-24 16:18:06 on dist-test-slave-v5j9"
server_key: "1de1e11b43c4dfc2c1d6f7b3c649930a"
server_key_iv: "cb9abe8524d103f74b16da7c37d607b4"
server_key_version: "encryptionkey@0"
I20260924 16:18:06.888113 21419 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 16:18:06.891764 21435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:18:06.893069 21419 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 16:18:06.893251 21419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266323169505-30632-0/minicluster-data/master-0/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266323169505-30632-0/minicluster-data/master-0/wal
uuid: "0559e481e0c544cfb875ac477e5a1d0a"
format_stamp: "Formatted at 2026-09-24 16:18:06 on dist-test-slave-v5j9"
server_key: "1de1e11b43c4dfc2c1d6f7b3c649930a"
server_key_iv: "cb9abe8524d103f74b16da7c37d607b4"
server_key_version: "encryptionkey@0"
I20260924 16:18:06.893491 21419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266323169505-30632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266323169505-30632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266323169505-30632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:18:06.907137 21419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:18:06.911578 21419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:18:06.911880 21419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:18:06.930585 21419 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.62:37915
I20260924 16:18:06.930604 21486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.62:37915 every 8 connection(s)
I20260924 16:18:06.932017 21419 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266323169505-30632-0/minicluster-data/master-0/data/info.pb
I20260924 16:18:06.935838 21487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:18:06.936578 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 21419
I20260924 16:18:06.936771 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266323169505-30632-0/minicluster-data/master-0/wal/instance
I20260924 16:18:06.937263 30632 external_mini_cluster.cc:1428] Setting key 37cbcb3169eef5e8ebfcdd99ec63b920
I20260924 16:18:06.942822 21487 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.234.62" port: 37915 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 16:18:06 dist-test-slave-v5j9 krb5kdc[21406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266686, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.62@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.234.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.61@KRBTEST.COM" created.
I20260924 16:18:06.965806 21490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:06.946147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:37915 (local address 127.0.0.1:50046)
0924 16:18:06.947008 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0924 16:18:06.947068 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 16:18:06.949241 (+ 2173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:18:06.950153 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:18:06.950161 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:18:06.950895 (+ 734us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:18:06.951230 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:18:06.951248 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:18:06.952807 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:18:06.952813 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:18:06.953977 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:18:06.953990 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:18:06.954916 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:18:06.954922 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:18:06.955115 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:18:06.955776 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:18:06.955803 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:18:06.957948 (+ 2145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:18:06.961517 (+ 3569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:18:06.961531 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:18:06.961544 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:18:06.962017 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:18:06.962415 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:18:06.962421 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:18:06.962424 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:18:06.962587 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:18:06.963315 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:18:06.963322 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:18:06.963541 (+ 219us) client_negotiation.cc:769] Sending connection context
0924 16:18:06.963791 (+ 250us) client_negotiation.cc:240] Negotiation successful
0924 16:18:06.964390 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"mutex_wait_us":137,"thread_start_us":369,"threads_started":1}
I20260924 16:18:06.965827 21491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:06.946148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50046 (local address 127.29.234.62:37915)
0924 16:18:06.947832 (+ 1684us) server_negotiation.cc:206] Beginning negotiation
0924 16:18:06.947845 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 16:18:06.947896 (+ 51us) server_negotiation.cc:407] Connection header received
0924 16:18:06.949687 (+ 1791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:18:06.949693 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:18:06.949929 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:18:06.950139 (+ 210us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:18:06.951448 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:06.952658 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:06.954122 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:06.954679 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:06.954878 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:18:06.958174 (+ 3296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:18:06.958183 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:18:06.958196 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:18:06.958228 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:18:06.961388 (+ 3160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:18:06.962156 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:18:06.962168 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:18:06.962171 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:18:06.962309 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:18:06.962698 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:18:06.962708 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:18:06.962710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:18:06.963184 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:18:06.963378 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0924 16:18:06.963941 (+ 563us) server_negotiation.cc:299] Negotiation successful
0924 16:18:06.964278 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":1454,"thread_start_us":1234,"threads_started":1}
I20260924 16:18:06.969089 21487 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.234.61" port: 34023 } has no permanent_uuid. Determining permanent_uuid...
I20260924 16:18:06.970372 21490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:06.969896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.61:34023 (local address 127.0.0.1:56394)
0924 16:18:06.970138 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0924 16:18:06.970256 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.61:34023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260924 16:18:06.970709 21437 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.234.61:34023: connect: Connection refused (error 111)
I20260924 16:18:06.972309 21490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:06.971949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.61:34023 (local address 127.0.0.1:56398)
0924 16:18:06.972122 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 16:18:06.972186 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.61:34023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 16:18:06.972527 21487 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.234.61:34023: Network error: Client connection negotiation failed: client connection to 127.29.234.61:34023: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.61.keytab.
Entry for principal kudu/127.29.234.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.61.keytab.
I20260924 16:18:06.996335 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266323169505-30632-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266323169505-30632-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266323169505-30632-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266323169505-30632-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.61:34023
--webserver_interface=127.29.234.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.61.keytab
--principal=kudu/127.29.234.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:35043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.234.62:37915,127.29.234.61:34023,127.29.234.60:41183
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:18:07.021865 21487 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.234.61" port: 34023 } attempt: 1
I20260924 16:18:07.023270 21490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:07.022676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.61:34023 (local address 127.0.0.1:56408)
0924 16:18:07.022920 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0924 16:18:07.023079 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.61:34023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260924 16:18:07.024343 21490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:07.023941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.61:34023 (local address 127.0.0.1:56418)
0924 16:18:07.024097 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 16:18:07.024191 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.61:34023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 16:18:07.024551 21487 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.234.61:34023: Network error: Client connection negotiation failed: client connection to 127.29.234.61:34023: connect: Connection refused (error 111)
I20260924 16:18:07.092828 21487 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.234.61" port: 34023 } attempt: 2
I20260924 16:18:07.094240 21490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:07.093666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.61:34023 (local address 127.0.0.1:56430)
0924 16:18:07.093939 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0924 16:18:07.094054 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.61:34023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260924 16:18:07.095376 21490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:07.094950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.61:34023 (local address 127.0.0.1:56440)
0924 16:18:07.095108 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 16:18:07.095217 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.61:34023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260924 16:18:07.095611 21487 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.234.61:34023: Network error: Client connection negotiation failed: client connection to 127.29.234.61:34023: connect: Connection refused (error 111)
I20260924 16:18:07.186903 21487 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.234.61" port: 34023 } attempt: 3
I20260924 16:18:07.188349 21490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:07.187700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.61:34023 (local address 127.0.0.1:56442)
0924 16:18:07.187970 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0924 16:18:07.188113 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.61:34023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260924 16:18:07.189638 21490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:07.189129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.61:34023 (local address 127.0.0.1:56454)
0924 16:18:07.189318 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 16:18:07.189463 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.61:34023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260924 16:18:07.189850 21487 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.234.61:34023: Network error: Client connection negotiation failed: client connection to 127.29.234.61:34023: connect: Connection refused (error 111)
I20260924 16:18:07.333101 21487 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.234.61" port: 34023 } attempt: 4
I20260924 16:18:07.334441 21490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:07.333811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.61:34023 (local address 127.0.0.1:56462)
0924 16:18:07.334153 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0924 16:18:07.334263 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.61:34023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166}
I20260924 16:18:07.335599 21490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:07.335191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.61:34023 (local address 127.0.0.1:56478)
0924 16:18:07.335352 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 16:18:07.335448 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.61:34023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260924 16:18:07.335830 21487 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.234.61:34023: Network error: Client connection negotiation failed: client connection to 127.29.234.61:34023: connect: Connection refused (error 111)
W20260924 16:18:07.340152 21494 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 16:18:07.340585 21494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:18:07.340687 21494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:18:07.340826 21494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:18:07.348528 21494 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:18:07.348623 21494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:18:07.348726 21494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:18:07.348811 21494 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:18:07.348898 21494 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:18:07.359157 21494 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:35043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266323169505-30632-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266323169505-30632-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.29.234.62:37915,127.29.234.61:34023,127.29.234.60:41183
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.61.keytab
--principal=kudu/127.29.234.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.61:34023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266323169505-30632-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.29.234.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266323169505-30632-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:18:07.360505 21494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:18:07.361966 21494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:18:07.374722 21499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:18:07.374756 21500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:18:07.375411 21504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:18:07.375710 21494 server_base.cc:1034] running on GCE node
I20260924 16:18:07.376531 21494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:18:07.378103 21494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:18:07.379354 21494 hybrid_clock.cc:648] HybridClock initialized: now 1790266687379274 us; error 41 us; skew 500 ppm
I20260924 16:18:07.379747 21494 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:18:07 dist-test-slave-v5j9 krb5kdc[21406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266687, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:18:07.383345 21494 init.cc:377] Logged in from keytab as kudu/127.29.234.61@KRBTEST.COM (short username kudu)
I20260924 16:18:07.384913 21494 webserver.cc:460] Webserver started at http://127.29.234.61:40599/ using document root <none> and password file <none>
I20260924 16:18:07.385795 21494 fs_manager.cc:359] Metadata directory not provided
I20260924 16:18:07.385937 21494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:18:07.386304 21494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:18:07.389736 21494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266323169505-30632-0/minicluster-data/master-1/data/instance:
uuid: "6bdf138dce19469eb482f094cb9b882a"
format_stamp: "Formatted at 2026-09-24 16:18:07 on dist-test-slave-v5j9"
server_key: "578f602d9819180f107c8d8263201676"
server_key_iv: "4dfde03c5e2eead5ab04b1683d6b3317"
server_key_version: "encryptionkey@0"
I20260924 16:18:07.390640 21494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266323169505-30632-0/minicluster-data/master-1/wal/instance:
uuid: "6bdf138dce19469eb482f094cb9b882a"
format_stamp: "Formatted at 2026-09-24 16:18:07 on dist-test-slave-v5j9"
server_key: "578f602d9819180f107c8d8263201676"
server_key_iv: "4dfde03c5e2eead5ab04b1683d6b3317"
server_key_version: "encryptionkey@0"
I20260924 16:18:07.396463 21494 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 16:18:07.400583 21511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:18:07.402220 21494 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 16:18:07.402426 21494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266323169505-30632-0/minicluster-data/master-1/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266323169505-30632-0/minicluster-data/master-1/wal
uuid: "6bdf138dce19469eb482f094cb9b882a"
format_stamp: "Formatted at 2026-09-24 16:18:07 on dist-test-slave-v5j9"
server_key: "578f602d9819180f107c8d8263201676"
server_key_iv: "4dfde03c5e2eead5ab04b1683d6b3317"
server_key_version: "encryptionkey@0"
I20260924 16:18:07.402674 21494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266323169505-30632-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266323169505-30632-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266323169505-30632-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:18:07.419004 21494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:18:07.423343 21494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:18:07.423625 21494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:18:07.442374 21494 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.61:34023
I20260924 16:18:07.442395 21563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.61:34023 every 8 connection(s)
I20260924 16:18:07.443897 21494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266323169505-30632-0/minicluster-data/master-1/data/info.pb
I20260924 16:18:07.447777 21564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:18:07.454138 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 21494
I20260924 16:18:07.454335 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266323169505-30632-0/minicluster-data/master-1/wal/instance
I20260924 16:18:07.454792 30632 external_mini_cluster.cc:1428] Setting key 7da54a07b23332253a56a7a8490a3c5c
I20260924 16:18:07.454373 21564 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.234.62" port: 37915 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 16:18:07 dist-test-slave-v5j9 krb5kdc[21406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266687, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.61@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:18:07.476240 21491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:07.456966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50050 (local address 127.29.234.62:37915)
0924 16:18:07.457218 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0924 16:18:07.457226 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:18:07.458454 (+ 1228us) server_negotiation.cc:407] Connection header received
0924 16:18:07.460480 (+ 2026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:18:07.460488 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:18:07.460584 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:18:07.460700 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:18:07.462266 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:07.463126 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:07.465634 (+ 2508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:07.466045 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:07.466199 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:18:07.470464 (+ 4265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:18:07.470470 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:18:07.470473 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:18:07.470507 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:18:07.473214 (+ 2707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:18:07.473992 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:18:07.473996 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:18:07.473999 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:18:07.474122 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:18:07.474528 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:18:07.474532 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:18:07.474534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:18:07.474819 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:18:07.474974 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 16:18:07.475675 (+ 701us) server_negotiation.cc:299] Negotiation successful
0924 16:18:07.475900 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 16:18:07.477213 21566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:07.457491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:37915 (local address 127.0.0.1:50050)
0924 16:18:07.458270 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0924 16:18:07.458325 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 16:18:07.460208 (+ 1883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:18:07.460902 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:18:07.460910 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:18:07.461544 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:18:07.462091 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:18:07.462108 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:18:07.463281 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:18:07.463288 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:18:07.465474 (+ 2186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:18:07.465493 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:18:07.466326 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:18:07.466336 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:18:07.466693 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:18:07.467453 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:18:07.467487 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:18:07.470226 (+ 2739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:18:07.473386 (+ 3160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:18:07.473395 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:18:07.473413 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:18:07.473856 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:18:07.474231 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:18:07.474236 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:18:07.474240 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:18:07.474410 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:18:07.474948 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:18:07.474956 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:18:07.475306 (+ 350us) client_negotiation.cc:769] Sending connection context
0924 16:18:07.475562 (+ 256us) client_negotiation.cc:240] Negotiation successful
0924 16:18:07.476080 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":320,"threads_started":1}
I20260924 16:18:07.479095 21564 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.234.61" port: 34023 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.29.234.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.234.60@KRBTEST.COM" created.
Sep 24 16:18:07 dist-test-slave-v5j9 krb5kdc[21406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266687, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.61@KRBTEST.COM for kudu/127.29.234.61@KRBTEST.COM
I20260924 16:18:07.493912 21566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:07.479943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.61:34023 (local address 127.0.0.1:56492)
0924 16:18:07.480120 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 16:18:07.480138 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 16:18:07.480246 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:18:07.481150 (+ 904us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:18:07.481154 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:18:07.481604 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:18:07.481728 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:18:07.481736 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:18:07.483161 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:18:07.483167 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:18:07.483984 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:18:07.483994 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:18:07.484778 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:18:07.484783 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:18:07.484996 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:18:07.485583 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:18:07.485612 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:18:07.488109 (+ 2497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:18:07.491491 (+ 3382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:18:07.491499 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:18:07.491503 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:18:07.491874 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:18:07.492265 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:18:07.492272 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:18:07.492277 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:18:07.492476 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:18:07.493054 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:18:07.493060 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:18:07.493237 (+ 177us) client_negotiation.cc:769] Sending connection context
0924 16:18:07.493439 (+ 202us) client_negotiation.cc:240] Negotiation successful
0924 16:18:07.493640 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66,"mutex_wait_us":36}
I20260924 16:18:07.494638 21568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:07.480172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56492 (local address 127.29.234.61:34023)
0924 16:18:07.480764 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0924 16:18:07.480770 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:18:07.480793 (+ 23us) server_negotiation.cc:407] Connection header received
0924 16:18:07.480836 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:18:07.480842 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:18:07.481004 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:18:07.481160 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:18:07.481906 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:07.483042 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:07.484119 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:07.484643 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:07.484771 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:18:07.488325 (+ 3554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:18:07.488335 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:18:07.488349 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:18:07.488376 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:18:07.491333 (+ 2957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:18:07.491993 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:18:07.491999 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:18:07.492002 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:18:07.492148 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:18:07.492593 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:18:07.492596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:18:07.492598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:18:07.492919 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:18:07.493072 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 16:18:07.493661 (+ 589us) server_negotiation.cc:299] Negotiation successful
0924 16:18:07.493954 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":459,"thread_start_us":326,"threads_started":1}
I20260924 16:18:07.497473 21564 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.234.60" port: 41183 } has no permanent_uuid. Determining permanent_uuid...
I20260924 16:18:07.498859 21566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:07.498341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.60:41183 (local address 127.0.0.1:35322)
0924 16:18:07.498588 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0924 16:18:07.498713 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.60:41183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260924 16:18:07.499119 21512 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.234.60:41183: connect: Connection refused (error 111)
I20260924 16:18:07.500746 21566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:07.500392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.60:41183 (local address 127.0.0.1:35336)
0924 16:18:07.500568 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 16:18:07.500643 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.60:41183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 16:18:07.500967 21564 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.234.60:41183: Network error: Client connection negotiation failed: client connection to 127.29.234.60:41183: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.234.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.60.keytab.
Entry for principal kudu/127.29.234.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.60.keytab.
I20260924 16:18:07.521541 30632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQurfSc/build/asan/bin/kudu
/tmp/dist-test-taskQurfSc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266323169505-30632-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266323169505-30632-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266323169505-30632-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266323169505-30632-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.60:41183
--webserver_interface=127.29.234.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.60.keytab
--principal=kudu/127.29.234.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.234.20:35043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.234.62:37915,127.29.234.61:34023,127.29.234.60:41183
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266323169505-30632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266323169505-30632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266323169505-30632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 16:18:07.550309 21564 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.234.60" port: 41183 } attempt: 1
I20260924 16:18:07.551626 21566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:07.551117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.60:41183 (local address 127.0.0.1:35346)
0924 16:18:07.551346 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0924 16:18:07.551463 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.60:41183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260924 16:18:07.552686 21566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:07.552318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.60:41183 (local address 127.0.0.1:35356)
0924 16:18:07.552498 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 16:18:07.552577 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.60:41183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 16:18:07.552887 21564 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.234.60:41183: Network error: Client connection negotiation failed: client connection to 127.29.234.60:41183: connect: Connection refused (error 111)
I20260924 16:18:07.621111 21564 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.234.60" port: 41183 } attempt: 2
I20260924 16:18:07.622470 21566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:07.621838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.60:41183 (local address 127.0.0.1:35358)
0924 16:18:07.622107 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0924 16:18:07.622259 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.60:41183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260924 16:18:07.623670 21566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:07.623187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.60:41183 (local address 127.0.0.1:35370)
0924 16:18:07.623376 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 16:18:07.623478 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.60:41183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260924 16:18:07.623922 21564 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.234.60:41183: Network error: Client connection negotiation failed: client connection to 127.29.234.60:41183: connect: Connection refused (error 111)
I20260924 16:18:07.635123 21487 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.234.61" port: 34023 } attempt: 5
Sep 24 16:18:07 dist-test-slave-v5j9 krb5kdc[21406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266686, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.62@KRBTEST.COM for kudu/127.29.234.61@KRBTEST.COM
I20260924 16:18:07.650955 21490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:07.635944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.61:34023 (local address 127.0.0.1:56506)
0924 16:18:07.636151 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0924 16:18:07.636177 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 16:18:07.636346 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:18:07.636755 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:18:07.636762 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:18:07.637205 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:18:07.637505 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:18:07.637517 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:18:07.638622 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:18:07.638628 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:18:07.639580 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:18:07.639592 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:18:07.640429 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:18:07.640436 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:18:07.640694 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:18:07.641470 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:18:07.641491 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:18:07.643878 (+ 2387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:18:07.648375 (+ 4497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:18:07.648381 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:18:07.648384 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:18:07.648834 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:18:07.649262 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:18:07.649266 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:18:07.649268 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:18:07.649480 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:18:07.650139 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:18:07.650143 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:18:07.650294 (+ 151us) client_negotiation.cc:769] Sending connection context
0924 16:18:07.650420 (+ 126us) client_negotiation.cc:240] Negotiation successful
0924 16:18:07.650680 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260924 16:18:07.651126 21568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:07.636162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56506 (local address 127.29.234.61:34023)
0924 16:18:07.636393 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0924 16:18:07.636402 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:18:07.636429 (+ 27us) server_negotiation.cc:407] Connection header received
0924 16:18:07.636524 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:18:07.636528 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:18:07.636597 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:18:07.636714 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:18:07.637678 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:07.638461 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:07.639747 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:07.640262 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:07.640419 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:18:07.644144 (+ 3725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:18:07.644154 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:18:07.644159 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:18:07.644220 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:18:07.648189 (+ 3969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:18:07.648971 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:18:07.648977 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:18:07.648980 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:18:07.649139 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:18:07.649605 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:18:07.649610 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:18:07.649613 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:18:07.649991 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:18:07.650144 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 16:18:07.650593 (+ 449us) server_negotiation.cc:299] Negotiation successful
0924 16:18:07.650830 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 16:18:07.652542 21487 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.234.60" port: 41183 } has no permanent_uuid. Determining permanent_uuid...
I20260924 16:18:07.653879 21490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:07.653439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.60:41183 (local address 127.0.0.1:35382)
0924 16:18:07.653642 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0924 16:18:07.653730 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.60:41183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260924 16:18:07.654982 21490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:07.654643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.60:41183 (local address 127.0.0.1:35384)
0924 16:18:07.654791 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 16:18:07.654880 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.60:41183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260924 16:18:07.655246 21487 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.234.60:41183: Network error: Client connection negotiation failed: client connection to 127.29.234.60:41183: connect: Connection refused (error 111)
I20260924 16:18:07.706507 21487 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.234.60" port: 41183 } attempt: 1
I20260924 16:18:07.707969 21490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:07.707392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.60:41183 (local address 127.0.0.1:35398)
0924 16:18:07.707653 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0924 16:18:07.707750 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.60:41183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260924 16:18:07.709115 21490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:07.708780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.60:41183 (local address 127.0.0.1:35414)
0924 16:18:07.708931 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 16:18:07.709003 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.60:41183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260924 16:18:07.709435 21487 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.234.60:41183: Network error: Client connection negotiation failed: client connection to 127.29.234.60:41183: connect: Connection refused (error 111)
I20260924 16:18:07.715157 21564 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.234.60" port: 41183 } attempt: 3
I20260924 16:18:07.716370 21566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:07.715813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.60:41183 (local address 127.0.0.1:35422)
0924 16:18:07.716019 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0924 16:18:07.716136 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.60:41183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260924 16:18:07.717681 21566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:07.717062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.60:41183 (local address 127.0.0.1:35438)
0924 16:18:07.717374 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0924 16:18:07.717491 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.60:41183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154}
W20260924 16:18:07.717916 21564 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.234.60:41183: Network error: Client connection negotiation failed: client connection to 127.29.234.60:41183: connect: Connection refused (error 111)
I20260924 16:18:07.777694 21487 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.234.60" port: 41183 } attempt: 2
I20260924 16:18:07.778990 21490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:07.778491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.60:41183 (local address 127.0.0.1:35446)
0924 16:18:07.778736 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0924 16:18:07.778828 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.60:41183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260924 16:18:07.780016 21490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:07.779718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.60:41183 (local address 127.0.0.1:35462)
0924 16:18:07.779844 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 16:18:07.779907 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.60:41183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 16:18:07.780223 21487 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.234.60:41183: Network error: Client connection negotiation failed: client connection to 127.29.234.60:41183: connect: Connection refused (error 111)
I20260924 16:18:07.861164 21564 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.234.60" port: 41183 } attempt: 4
I20260924 16:18:07.862447 21566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:07.861856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.60:41183 (local address 127.0.0.1:35476)
0924 16:18:07.862117 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0924 16:18:07.862244 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.60:41183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260924 16:18:07.863510 21566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:07.863133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.60:41183 (local address 127.0.0.1:35480)
0924 16:18:07.863297 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 16:18:07.863380 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.60:41183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260924 16:18:07.863731 21564 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.234.60:41183: Network error: Client connection negotiation failed: client connection to 127.29.234.60:41183: connect: Connection refused (error 111)
W20260924 16:18:07.877377 21571 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 16:18:07.877767 21571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 16:18:07.877877 21571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 16:18:07.878005 21571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 16:18:07.885186 21571 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 16:18:07.885268 21571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 16:18:07.885380 21571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 16:18:07.885468 21571 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 16:18:07.885577 21571 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 16:18:07.886464 21487 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.234.60" port: 41183 } attempt: 3
I20260924 16:18:07.887603 21490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:07.887112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.60:41183 (local address 127.0.0.1:35486)
0924 16:18:07.887361 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0924 16:18:07.887446 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.60:41183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260924 16:18:07.888665 21490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:07.888257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.60:41183 (local address 127.0.0.1:35488)
0924 16:18:07.888485 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0924 16:18:07.888568 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.234.60:41183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260924 16:18:07.888868 21487 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.234.60:41183: Network error: Client connection negotiation failed: client connection to 127.29.234.60:41183: connect: Connection refused (error 111)
I20260924 16:18:07.896019 21571 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.234.20:35043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266323169505-30632-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266323169505-30632-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.29.234.62:37915,127.29.234.61:34023,127.29.234.60:41183
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266323169505-30632-0/krb5kdc/kudu_127.29.234.60.keytab
--principal=kudu/127.29.234.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.234.60:41183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266323169505-30632-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.29.234.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266323169505-30632-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 15:07:34 UTC on ccycloud.quasar-oatmms.root.comops.site
build id 4670396
ASAN enabled
I20260924 16:18:07.897392 21571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 16:18:07.898802 21571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 16:18:07.910992 21576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:18:07.911027 21577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 16:18:07.910943 21580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 16:18:07.911875 21571 server_base.cc:1034] running on GCE node
I20260924 16:18:07.912613 21571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 16:18:07.914064 21571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 16:18:07.915333 21571 hybrid_clock.cc:648] HybridClock initialized: now 1790266687915246 us; error 56 us; skew 500 ppm
I20260924 16:18:07.915706 21571 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 16:18:07 dist-test-slave-v5j9 krb5kdc[21406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266687, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 16:18:07.918848 21571 init.cc:377] Logged in from keytab as kudu/127.29.234.60@KRBTEST.COM (short username kudu)
I20260924 16:18:07.920459 21571 webserver.cc:460] Webserver started at http://127.29.234.60:39253/ using document root <none> and password file <none>
I20260924 16:18:07.921204 21571 fs_manager.cc:359] Metadata directory not provided
I20260924 16:18:07.921337 21571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 16:18:07.921749 21571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 16:18:07.924831 21571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266323169505-30632-0/minicluster-data/master-2/data/instance:
uuid: "dbdf9aa2862a4a9093a60043d7f07d95"
format_stamp: "Formatted at 2026-09-24 16:18:07 on dist-test-slave-v5j9"
server_key: "ae095f0ac693add1032e8cbd9353208a"
server_key_iv: "a9f6c35d08ad4db799e39e1db21fb266"
server_key_version: "encryptionkey@0"
I20260924 16:18:07.925742 21571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266323169505-30632-0/minicluster-data/master-2/wal/instance:
uuid: "dbdf9aa2862a4a9093a60043d7f07d95"
format_stamp: "Formatted at 2026-09-24 16:18:07 on dist-test-slave-v5j9"
server_key: "ae095f0ac693add1032e8cbd9353208a"
server_key_iv: "a9f6c35d08ad4db799e39e1db21fb266"
server_key_version: "encryptionkey@0"
I20260924 16:18:07.931188 21571 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 16:18:07.934792 21587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 16:18:07.936282 21571 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 16:18:07.936457 21571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266323169505-30632-0/minicluster-data/master-2/data,/tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266323169505-30632-0/minicluster-data/master-2/wal
uuid: "dbdf9aa2862a4a9093a60043d7f07d95"
format_stamp: "Formatted at 2026-09-24 16:18:07 on dist-test-slave-v5j9"
server_key: "ae095f0ac693add1032e8cbd9353208a"
server_key_iv: "a9f6c35d08ad4db799e39e1db21fb266"
server_key_version: "encryptionkey@0"
I20260924 16:18:07.936677 21571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266323169505-30632-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266323169505-30632-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266323169505-30632-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 16:18:07.961894 21571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 16:18:07.966214 21571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 16:18:07.966522 21571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 16:18:07.985488 21571 rpc_server.cc:307] RPC server started. Bound to: 127.29.234.60:41183
I20260924 16:18:07.985500 21638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.234.60:41183 every 8 connection(s)
I20260924 16:18:07.986943 21571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266323169505-30632-0/minicluster-data/master-2/data/info.pb
I20260924 16:18:07.989248 30632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQurfSc/build/asan/bin/kudu as pid 21571
I20260924 16:18:07.989490 30632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQurfSc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790266323169505-30632-0/minicluster-data/master-2/wal/instance
I20260924 16:18:07.989946 30632 external_mini_cluster.cc:1428] Setting key 84237520ecb987fb2904a697b9790aa0
I20260924 16:18:07.991449 21639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 16:18:08.001848 21639 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.234.62" port: 37915 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 16:18:08 dist-test-slave-v5j9 krb5kdc[21406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266686, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.234.60@KRBTEST.COM
Sep 24 16:18:08 dist-test-slave-v5j9 krb5kdc[21406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266687, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.60@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
I20260924 16:18:08.016263 21642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:07.992082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35496 (local address 127.29.234.60:41183)
0924 16:18:07.993051 (+ 969us) server_negotiation.cc:206] Beginning negotiation
0924 16:18:07.993070 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 16:18:07.993537 (+ 467us) server_negotiation.cc:407] Connection header received
0924 16:18:07.995268 (+ 1731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:18:07.995299 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:18:07.995881 (+ 582us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:18:07.996349 (+ 468us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:18:07.997862 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:07.999931 (+ 2069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:08.001275 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:08.002390 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:08.002599 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:18:08.007574 (+ 4975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:18:08.007585 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:18:08.007600 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:18:08.007646 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:18:08.011412 (+ 3766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:18:08.012313 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:18:08.012324 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:18:08.012338 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:18:08.012561 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:18:08.013000 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:18:08.013009 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:18:08.013012 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:18:08.013819 (+ 807us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:18:08.014062 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0924 16:18:08.014749 (+ 687us) server_negotiation.cc:299] Negotiation successful
0924 16:18:08.015226 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":709,"thread_start_us":409,"threads_started":1}
I20260924 16:18:08.022116 21644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:08.004934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.62:37915 (local address 127.0.0.1:50064)
0924 16:18:08.005852 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0924 16:18:08.005905 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 16:18:08.006141 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:18:08.006622 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:18:08.006633 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:18:08.007455 (+ 822us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:18:08.007854 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:18:08.007874 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:18:08.008959 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:18:08.008969 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:18:08.010811 (+ 1842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:18:08.010827 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:18:08.011653 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:18:08.011658 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:18:08.011787 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:18:08.012433 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:18:08.012469 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:18:08.015340 (+ 2871us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:18:08.019153 (+ 3813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:18:08.019167 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:18:08.019176 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:18:08.019863 (+ 687us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:18:08.020296 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:18:08.020301 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:18:08.020304 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:18:08.020472 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:18:08.021003 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:18:08.021014 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:18:08.021263 (+ 249us) client_negotiation.cc:769] Sending connection context
0924 16:18:08.021494 (+ 231us) client_negotiation.cc:240] Negotiation successful
0924 16:18:08.021819 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":502,"threads_started":1}
I20260924 16:18:08.022439 21643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:08.004869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50064 (local address 127.29.234.62:37915)
0924 16:18:08.005457 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0924 16:18:08.005464 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:18:08.006020 (+ 556us) server_negotiation.cc:407] Connection header received
0924 16:18:08.006285 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:18:08.006290 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:18:08.006359 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:18:08.006479 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:18:08.008047 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:08.008800 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:08.010984 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:08.011431 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:08.011615 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:18:08.015558 (+ 3943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:18:08.015564 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:18:08.015568 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:18:08.015610 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:18:08.018940 (+ 3330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:18:08.020001 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:18:08.020007 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:18:08.020010 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:18:08.020184 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:18:08.020599 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:18:08.020603 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:18:08.020605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:18:08.020866 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:18:08.020989 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 16:18:08.021642 (+ 653us) server_negotiation.cc:299] Negotiation successful
0924 16:18:08.021925 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":267,"threads_started":1}
I20260924 16:18:08.023840 21639 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.234.61" port: 34023 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 16:18:08 dist-test-slave-v5j9 krb5kdc[21406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266686, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.234.62@KRBTEST.COM
Sep 24 16:18:08 dist-test-slave-v5j9 krb5kdc[21406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266687, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.60@KRBTEST.COM for kudu/127.29.234.61@KRBTEST.COM
I20260924 16:18:08.037424 21644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:08.024570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.61:34023 (local address 127.0.0.1:56516)
0924 16:18:08.024740 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 16:18:08.024761 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 16:18:08.024857 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:18:08.025184 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:18:08.025188 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:18:08.025757 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:18:08.025953 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:18:08.025965 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:18:08.027140 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:18:08.027144 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:18:08.028021 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:18:08.028032 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:18:08.029057 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:18:08.029061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:18:08.029241 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:18:08.029857 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:18:08.029881 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:18:08.031894 (+ 2013us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:18:08.034956 (+ 3062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:18:08.034961 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:18:08.034965 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:18:08.035335 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:18:08.035746 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:18:08.035751 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:18:08.035755 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:18:08.035901 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:18:08.036416 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:18:08.036421 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:18:08.036581 (+ 160us) client_negotiation.cc:769] Sending connection context
0924 16:18:08.036723 (+ 142us) client_negotiation.cc:240] Negotiation successful
0924 16:18:08.037006 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260924 16:18:08.037634 21568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:08.024698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56516 (local address 127.29.234.61:34023)
0924 16:18:08.024859 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 16:18:08.024863 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 16:18:08.024878 (+ 15us) server_negotiation.cc:407] Connection header received
0924 16:18:08.024956 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:18:08.024960 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:18:08.025014 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:18:08.025089 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:18:08.026144 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:08.026973 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:08.028187 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:08.028894 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:08.029026 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:18:08.032092 (+ 3066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:18:08.032097 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:18:08.032101 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:18:08.032134 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:18:08.034805 (+ 2671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:18:08.035464 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:18:08.035470 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:18:08.035475 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:18:08.035630 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:18:08.036025 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:18:08.036035 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:18:08.036037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:18:08.036297 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:18:08.036450 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 16:18:08.036815 (+ 365us) server_negotiation.cc:299] Negotiation successful
0924 16:18:08.037020 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 16:18:08.037505 21645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:08.019325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50080 (local address 127.29.234.62:37915)
0924 16:18:08.019980 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0924 16:18:08.019985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:18:08.020006 (+ 21us) server_negotiation.cc:407] Connection header received
0924 16:18:08.020048 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:18:08.020054 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:18:08.020143 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:18:08.020258 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:18:08.021837 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:08.023113 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:08.024487 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:08.025198 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:08.025321 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:18:08.031111 (+ 5790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:18:08.031117 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:18:08.031120 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:18:08.031155 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:18:08.034728 (+ 3573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:18:08.035428 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:18:08.035433 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:18:08.035436 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:18:08.035565 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:18:08.035955 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:18:08.035960 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:18:08.035962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:18:08.036252 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:18:08.036526 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0924 16:18:08.036829 (+ 303us) server_negotiation.cc:299] Negotiation successful
0924 16:18:08.037049 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":294,"threads_started":1}
I20260924 16:18:08.038480 21639 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.234.60" port: 41183 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 16:18:08 dist-test-slave-v5j9 krb5kdc[21406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266687, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.60@KRBTEST.COM for kudu/127.29.234.60@KRBTEST.COM
Sep 24 16:18:08 dist-test-slave-v5j9 krb5kdc[21406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266686, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.234.61@KRBTEST.COM
I20260924 16:18:08.053475 21642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:08.039438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35506 (local address 127.29.234.60:41183)
0924 16:18:08.039583 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 16:18:08.039589 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:18:08.039606 (+ 17us) server_negotiation.cc:407] Connection header received
0924 16:18:08.039655 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:18:08.039662 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:18:08.039741 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:18:08.039808 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:18:08.040671 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:08.041786 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:08.043369 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:08.043976 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:08.044106 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:18:08.047634 (+ 3528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:18:08.047642 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:18:08.047648 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:18:08.047692 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:18:08.050855 (+ 3163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:18:08.051496 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:18:08.051502 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:18:08.051506 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:18:08.051631 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:18:08.051981 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:18:08.051988 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:18:08.051991 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:18:08.052280 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:18:08.052431 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 16:18:08.052776 (+ 345us) server_negotiation.cc:299] Negotiation successful
0924 16:18:08.053012 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 16:18:08.053553 21644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:08.039221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.60:41183 (local address 127.0.0.1:35506)
0924 16:18:08.039379 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 16:18:08.039397 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 16:18:08.039540 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:18:08.039840 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:18:08.039845 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:18:08.040322 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:18:08.040518 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:18:08.040532 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:18:08.041916 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:18:08.041922 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:18:08.043236 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:18:08.043248 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:18:08.044168 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:18:08.044173 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:18:08.044341 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:18:08.044939 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:18:08.044964 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:18:08.047430 (+ 2466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:18:08.051006 (+ 3576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:18:08.051012 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:18:08.051016 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:18:08.051376 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:18:08.051740 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:18:08.051746 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:18:08.051751 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:18:08.051878 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:18:08.052393 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:18:08.052398 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:18:08.052542 (+ 144us) client_negotiation.cc:769] Sending connection context
0924 16:18:08.052699 (+ 157us) client_negotiation.cc:240] Negotiation successful
0924 16:18:08.053171 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":151}
I20260924 16:18:08.056118 21568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:08.039271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56530 (local address 127.29.234.61:34023)
0924 16:18:08.039418 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 16:18:08.039424 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:18:08.039564 (+ 140us) server_negotiation.cc:407] Connection header received
0924 16:18:08.039742 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:18:08.039748 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:18:08.039818 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:18:08.039930 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:18:08.041774 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:08.042887 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:08.044489 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:08.045145 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:08.045279 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:18:08.050358 (+ 5079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:18:08.050364 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:18:08.050368 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:18:08.050404 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:18:08.053091 (+ 2687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:18:08.054062 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:18:08.054068 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:18:08.054074 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:18:08.054263 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:18:08.054682 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:18:08.054688 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:18:08.054691 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:18:08.054971 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:18:08.055260 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0924 16:18:08.055545 (+ 285us) server_negotiation.cc:299] Negotiation successful
0924 16:18:08.055781 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 16:18:08.059362 21639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbdf9aa2862a4a9093a60043d7f07d95: Bootstrap starting.
I20260924 16:18:08.061324 30632 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 16:18:08.063048 21639 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dbdf9aa2862a4a9093a60043d7f07d95: Neither blocks nor log segments found. Creating new log.
I20260924 16:18:08.064464 21639 log.cc:824] T 00000000000000000000000000000000 P dbdf9aa2862a4a9093a60043d7f07d95: Log is configured to *not* fsync() on all Append() calls
I20260924 16:18:08.066123 21487 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.234.60" port: 41183 } attempt: 4
I20260924 16:18:08.063796 21462 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "0559e481e0c544cfb875ac477e5a1d0a"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50080
I20260924 16:18:08.068643 21639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbdf9aa2862a4a9093a60043d7f07d95: No bootstrap required, opened a new log
Sep 24 16:18:08 dist-test-slave-v5j9 krb5kdc[21406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266686, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.62@KRBTEST.COM for kudu/127.29.234.60@KRBTEST.COM
I20260924 16:18:08.078970 21639 raft_consensus.cc:348] T 00000000000000000000000000000000 P dbdf9aa2862a4a9093a60043d7f07d95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0559e481e0c544cfb875ac477e5a1d0a" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 37915 } } peers { permanent_uuid: "6bdf138dce19469eb482f094cb9b882a" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 34023 } } peers { permanent_uuid: "dbdf9aa2862a4a9093a60043d7f07d95" member_type: VOTER last_known_addr { host: "127.29.234.60" port: 41183 } }
I20260924 16:18:08.079425 21639 raft_consensus.cc:729] T 00000000000000000000000000000000 P dbdf9aa2862a4a9093a60043d7f07d95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbdf9aa2862a4a9093a60043d7f07d95, State: Initialized, Role: FOLLOWER
I20260924 16:18:08.080018 21639 consensus_queue.cc:260] T 00000000000000000000000000000000 P dbdf9aa2862a4a9093a60043d7f07d95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0559e481e0c544cfb875ac477e5a1d0a" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 37915 } } peers { permanent_uuid: "6bdf138dce19469eb482f094cb9b882a" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 34023 } } peers { permanent_uuid: "dbdf9aa2862a4a9093a60043d7f07d95" member_type: VOTER last_known_addr { host: "127.29.234.60" port: 41183 } }
I20260924 16:18:08.081043 21647 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbdf9aa2862a4a9093a60043d7f07d95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0559e481e0c544cfb875ac477e5a1d0a" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 37915 } } peers { permanent_uuid: "6bdf138dce19469eb482f094cb9b882a" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 34023 } } peers { permanent_uuid: "dbdf9aa2862a4a9093a60043d7f07d95" member_type: VOTER last_known_addr { host: "127.29.234.60" port: 41183 } } }
I20260924 16:18:08.081544 21647 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbdf9aa2862a4a9093a60043d7f07d95 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:18:08.081575 21642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:08.066955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35522 (local address 127.29.234.60:41183)
0924 16:18:08.067132 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 16:18:08.067137 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:18:08.067190 (+ 53us) server_negotiation.cc:407] Connection header received
0924 16:18:08.067409 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:18:08.067412 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:18:08.067472 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:18:08.067591 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:18:08.068462 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:08.069289 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:08.070543 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:08.071078 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:08.071230 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:18:08.074806 (+ 3576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:18:08.074814 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:18:08.074821 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:18:08.074856 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:18:08.078575 (+ 3719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:18:08.079380 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:18:08.079387 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:18:08.079393 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:18:08.079558 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:18:08.079957 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:18:08.079962 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:18:08.079966 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:18:08.080393 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:18:08.080535 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 16:18:08.080957 (+ 422us) server_negotiation.cc:299] Negotiation successful
0924 16:18:08.081155 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 16:18:08.081902 21490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:08.066836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.60:41183 (local address 127.0.0.1:35522)
0924 16:18:08.067053 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0924 16:18:08.067125 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0924 16:18:08.067279 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:18:08.067621 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:18:08.067626 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:18:08.068140 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:18:08.068313 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:18:08.068322 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:18:08.069472 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:18:08.069477 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:18:08.070400 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:18:08.070411 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:18:08.071234 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:18:08.071240 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:18:08.071443 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:18:08.072075 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:18:08.072093 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:18:08.074577 (+ 2484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:18:08.078729 (+ 4152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:18:08.078736 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:18:08.078740 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:18:08.079249 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:18:08.079661 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:18:08.079666 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:18:08.079669 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:18:08.079844 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:18:08.080504 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:18:08.080509 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:18:08.080679 (+ 170us) client_negotiation.cc:769] Sending connection context
0924 16:18:08.080835 (+ 156us) client_negotiation.cc:240] Negotiation successful
0924 16:18:08.081130 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260924 16:18:08.082509 21639 sys_catalog.cc:564] T 00000000000000000000000000000000 P dbdf9aa2862a4a9093a60043d7f07d95 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:18:08.087675 21487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0559e481e0c544cfb875ac477e5a1d0a: Bootstrap starting.
I20260924 16:18:08.091831 21487 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0559e481e0c544cfb875ac477e5a1d0a: Neither blocks nor log segments found. Creating new log.
W20260924 16:18:08.092640 21659 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dbdf9aa2862a4a9093a60043d7f07d95: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 16:18:08.092767 21659 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 16:18:08.093360 21487 log.cc:824] T 00000000000000000000000000000000 P 0559e481e0c544cfb875ac477e5a1d0a: Log is configured to *not* fsync() on all Append() calls
I20260924 16:18:08.097067 21487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0559e481e0c544cfb875ac477e5a1d0a: No bootstrap required, opened a new log
I20260924 16:18:08.090427 21571 master_runner.cc:186] Error getting master registration for 127.29.234.62:37915: OK, instance_id { permanent_uuid: "0559e481e0c544cfb875ac477e5a1d0a" instance_seqno: 1790266686925482 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 16:18:08.101816 21571 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 16:18:08.106137 21487 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0559e481e0c544cfb875ac477e5a1d0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0559e481e0c544cfb875ac477e5a1d0a" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 37915 } } peers { permanent_uuid: "6bdf138dce19469eb482f094cb9b882a" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 34023 } } peers { permanent_uuid: "dbdf9aa2862a4a9093a60043d7f07d95" member_type: VOTER last_known_addr { host: "127.29.234.60" port: 41183 } }
I20260924 16:18:08.106645 21487 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0559e481e0c544cfb875ac477e5a1d0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0559e481e0c544cfb875ac477e5a1d0a, State: Initialized, Role: FOLLOWER
I20260924 16:18:08.107228 21487 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0559e481e0c544cfb875ac477e5a1d0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0559e481e0c544cfb875ac477e5a1d0a" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 37915 } } peers { permanent_uuid: "6bdf138dce19469eb482f094cb9b882a" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 34023 } } peers { permanent_uuid: "dbdf9aa2862a4a9093a60043d7f07d95" member_type: VOTER last_known_addr { host: "127.29.234.60" port: 41183 } }
I20260924 16:18:08.109956 21662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0559e481e0c544cfb875ac477e5a1d0a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0559e481e0c544cfb875ac477e5a1d0a" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 37915 } } peers { permanent_uuid: "6bdf138dce19469eb482f094cb9b882a" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 34023 } } peers { permanent_uuid: "dbdf9aa2862a4a9093a60043d7f07d95" member_type: VOTER last_known_addr { host: "127.29.234.60" port: 41183 } } }
I20260924 16:18:08.110409 21662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0559e481e0c544cfb875ac477e5a1d0a [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:18:08.111433 21487 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0559e481e0c544cfb875ac477e5a1d0a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:18:08.111577 21462 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0559e481e0c544cfb875ac477e5a1d0a [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 16:18:08.111791 21462 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0559e481e0c544cfb875ac477e5a1d0a [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:18:08.114477 21462 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0559e481e0c544cfb875ac477e5a1d0a [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0559e481e0c544cfb875ac477e5a1d0a" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 37915 } } peers { permanent_uuid: "6bdf138dce19469eb482f094cb9b882a" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 34023 } } peers { permanent_uuid: "dbdf9aa2862a4a9093a60043d7f07d95" member_type: VOTER last_known_addr { host: "127.29.234.60" port: 41183 } }
I20260924 16:18:08.116655 21614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0559e481e0c544cfb875ac477e5a1d0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "dbdf9aa2862a4a9093a60043d7f07d95"
I20260924 16:18:08.117219 21614 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dbdf9aa2862a4a9093a60043d7f07d95 [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:18:08.120591 21614 raft_consensus.cc:2454] T 00000000000000000000000000000000 P dbdf9aa2862a4a9093a60043d7f07d95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0559e481e0c544cfb875ac477e5a1d0a in term 1.
I20260924 16:18:08.120950 21462 leader_election.cc:288] T 00000000000000000000000000000000 P 0559e481e0c544cfb875ac477e5a1d0a [CANDIDATE]: Term 1 election: Requested vote from peers 6bdf138dce19469eb482f094cb9b882a (127.29.234.61:34023), dbdf9aa2862a4a9093a60043d7f07d95 (127.29.234.60:41183)
I20260924 16:18:08.121250 21436 leader_election.cc:302] T 00000000000000000000000000000000 P 0559e481e0c544cfb875ac477e5a1d0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0559e481e0c544cfb875ac477e5a1d0a, dbdf9aa2862a4a9093a60043d7f07d95; no voters:
I20260924 16:18:08.121729 21662 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0559e481e0c544cfb875ac477e5a1d0a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 16:18:08.132772 21662 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0559e481e0c544cfb875ac477e5a1d0a [term 1 LEADER]: Becoming Leader. State: Replica: 0559e481e0c544cfb875ac477e5a1d0a, State: Running, Role: LEADER
I20260924 16:18:08.133932 21662 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0559e481e0c544cfb875ac477e5a1d0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0559e481e0c544cfb875ac477e5a1d0a" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 37915 } } peers { permanent_uuid: "6bdf138dce19469eb482f094cb9b882a" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 34023 } } peers { permanent_uuid: "dbdf9aa2862a4a9093a60043d7f07d95" member_type: VOTER last_known_addr { host: "127.29.234.60" port: 41183 } }
I20260924 16:18:08.148980 21666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0559e481e0c544cfb875ac477e5a1d0a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0559e481e0c544cfb875ac477e5a1d0a. Latest consensus state: current_term: 1 leader_uuid: "0559e481e0c544cfb875ac477e5a1d0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0559e481e0c544cfb875ac477e5a1d0a" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 37915 } } peers { permanent_uuid: "6bdf138dce19469eb482f094cb9b882a" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 34023 } } peers { permanent_uuid: "dbdf9aa2862a4a9093a60043d7f07d95" member_type: VOTER last_known_addr { host: "127.29.234.60" port: 41183 } } }
I20260924 16:18:08.149489 21666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0559e481e0c544cfb875ac477e5a1d0a [sys.catalog]: This master's current role is: LEADER
I20260924 16:18:08.159319 21676 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 16:18:08.162948 21564 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.234.60" port: 41183 } attempt: 5
W20260924 16:18:08.164803 21441 tablet.cc:2369] T 00000000000000000000000000000000 P 0559e481e0c544cfb875ac477e5a1d0a: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 16:18:08 dist-test-slave-v5j9 krb5kdc[21406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790266687, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.234.61@KRBTEST.COM for kudu/127.29.234.60@KRBTEST.COM
I20260924 16:18:08.190080 21642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:08.162146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35536 (local address 127.29.234.60:41183)
0924 16:18:08.162321 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 16:18:08.162329 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:18:08.169122 (+ 6793us) server_negotiation.cc:407] Connection header received
0924 16:18:08.169789 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:18:08.169795 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:18:08.169868 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:18:08.169933 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:18:08.172709 (+ 2776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:08.173832 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:08.176199 (+ 2367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:08.177092 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:08.177188 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:18:08.182359 (+ 5171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:18:08.182365 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:18:08.182372 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:18:08.182413 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:18:08.185892 (+ 3479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:18:08.186860 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:18:08.186865 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:18:08.186868 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:18:08.187044 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:18:08.187502 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:18:08.187511 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:18:08.187515 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:18:08.187971 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:18:08.188213 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0924 16:18:08.189446 (+ 1233us) server_negotiation.cc:299] Negotiation successful
0924 16:18:08.189704 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 16:18:08.193463 21676 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 16:18:08.194365 21566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:08.163673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.234.60:41183 (local address 127.0.0.1:35544)
0924 16:18:08.163862 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 16:18:08.163886 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 16:18:08.164030 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 16:18:08.167197 (+ 3167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 16:18:08.167213 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 16:18:08.167753 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0924 16:18:08.167975 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:18:08.167989 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:18:08.170696 (+ 2707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:18:08.170701 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:18:08.172308 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 16:18:08.172326 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 16:18:08.174125 (+ 1799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 16:18:08.174134 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 16:18:08.174357 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:18:08.175210 (+ 853us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 16:18:08.175235 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 16:18:08.179128 (+ 3893us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 16:18:08.191444 (+ 12316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:18:08.191449 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:18:08.191453 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:18:08.192020 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:18:08.192533 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 16:18:08.192539 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 16:18:08.192543 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 16:18:08.192712 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 16:18:08.193376 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 16:18:08.193381 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 16:18:08.193531 (+ 150us) client_negotiation.cc:769] Sending connection context
0924 16:18:08.193680 (+ 149us) client_negotiation.cc:240] Negotiation successful
0924 16:18:08.193986 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260924 16:18:08.195669 21681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:08.163759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35544 (local address 127.29.234.60:41183)
0924 16:18:08.166508 (+ 2749us) server_negotiation.cc:206] Beginning negotiation
0924 16:18:08.166520 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 16:18:08.166580 (+ 60us) server_negotiation.cc:407] Connection header received
0924 16:18:08.166725 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:18:08.166735 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:18:08.166939 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:18:08.167199 (+ 260us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:18:08.168202 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:08.170462 (+ 2260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:08.172696 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:08.173897 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:08.179432 (+ 5535us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:18:08.179922 (+ 490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:18:08.179931 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:18:08.179936 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:18:08.180007 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:18:08.191217 (+ 11210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:18:08.192214 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:18:08.192219 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:18:08.192222 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:18:08.192388 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:18:08.192864 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:18:08.192869 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:18:08.192872 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:18:08.193216 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:18:08.194879 (+ 1663us) server_negotiation.cc:1035] Waiting for connection context
0924 16:18:08.195005 (+ 126us) server_negotiation.cc:299] Negotiation successful
0924 16:18:08.195238 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":2409,"thread_start_us":495,"threads_started":1}
I20260924 16:18:08.201798 21564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bdf138dce19469eb482f094cb9b882a: Bootstrap starting.
I20260924 16:18:08.206843 21645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:08.161953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50094 (local address 127.29.234.62:37915)
0924 16:18:08.162317 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0924 16:18:08.162324 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:18:08.181979 (+ 19655us) server_negotiation.cc:407] Connection header received
0924 16:18:08.182313 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:18:08.182320 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:18:08.182406 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:18:08.182481 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:18:08.184597 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:08.185852 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:08.188006 (+ 2154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:08.188741 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:08.188838 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:18:08.199595 (+ 10757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:18:08.199602 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:18:08.199607 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:18:08.199649 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:18:08.203438 (+ 3789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:18:08.204344 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:18:08.204349 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:18:08.204353 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:18:08.204520 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:18:08.204981 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:18:08.204986 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:18:08.204989 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:18:08.205339 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:18:08.205592 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0924 16:18:08.206238 (+ 646us) server_negotiation.cc:299] Negotiation successful
0924 16:18:08.206463 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194}
I20260924 16:18:08.214263 21564 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6bdf138dce19469eb482f094cb9b882a: Neither blocks nor log segments found. Creating new log.
I20260924 16:18:08.215737 21564 log.cc:824] T 00000000000000000000000000000000 P 6bdf138dce19469eb482f094cb9b882a: Log is configured to *not* fsync() on all Append() calls
W20260924 16:18:08.218683 21593 tablet.cc:2369] T 00000000000000000000000000000000 P dbdf9aa2862a4a9093a60043d7f07d95: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 16:18:08.223549 21568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:08.161860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56546 (local address 127.29.234.61:34023)
0924 16:18:08.162110 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0924 16:18:08.162120 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:18:08.179462 (+ 17342us) server_negotiation.cc:407] Connection header received
0924 16:18:08.179840 (+ 378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:18:08.179846 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:18:08.179928 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:18:08.179997 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:18:08.187969 (+ 7972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:08.189084 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:08.207319 (+ 18235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:08.208015 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:08.208126 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:18:08.216040 (+ 7914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 16:18:08.216047 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 16:18:08.216052 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 16:18:08.216092 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 16:18:08.219719 (+ 3627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:18:08.221059 (+ 1340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:18:08.221065 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:18:08.221068 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:18:08.221226 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 16:18:08.221713 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 16:18:08.221720 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 16:18:08.221725 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 16:18:08.222098 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 16:18:08.222317 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0924 16:18:08.222913 (+ 596us) server_negotiation.cc:299] Negotiation successful
0924 16:18:08.223181 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 16:18:08.211699 21419 master_runner.cc:186] Error getting master registration for 127.29.234.61:34023: OK, instance_id { permanent_uuid: "6bdf138dce19469eb482f094cb9b882a" instance_seqno: 1790266687436913 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 16:18:08.224260 21419 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 16:18:08.229172 21564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bdf138dce19469eb482f094cb9b882a: No bootstrap required, opened a new log
I20260924 16:18:08.242631 21614 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dbdf9aa2862a4a9093a60043d7f07d95 [term 1 FOLLOWER]: Refusing update from remote peer 0559e481e0c544cfb875ac477e5a1d0a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:18:08.243772 21666 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0559e481e0c544cfb875ac477e5a1d0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbdf9aa2862a4a9093a60043d7f07d95" member_type: VOTER last_known_addr { host: "127.29.234.60" port: 41183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:18:08.256155 21564 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6bdf138dce19469eb482f094cb9b882a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0559e481e0c544cfb875ac477e5a1d0a" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 37915 } } peers { permanent_uuid: "6bdf138dce19469eb482f094cb9b882a" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 34023 } } peers { permanent_uuid: "dbdf9aa2862a4a9093a60043d7f07d95" member_type: VOTER last_known_addr { host: "127.29.234.60" port: 41183 } }
I20260924 16:18:08.262351 21564 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6bdf138dce19469eb482f094cb9b882a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bdf138dce19469eb482f094cb9b882a, State: Initialized, Role: FOLLOWER
I20260924 16:18:08.263185 21564 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6bdf138dce19469eb482f094cb9b882a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0559e481e0c544cfb875ac477e5a1d0a" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 37915 } } peers { permanent_uuid: "6bdf138dce19469eb482f094cb9b882a" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 34023 } } peers { permanent_uuid: "dbdf9aa2862a4a9093a60043d7f07d95" member_type: VOTER last_known_addr { host: "127.29.234.60" port: 41183 } }
I20260924 16:18:08.267838 21647 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbdf9aa2862a4a9093a60043d7f07d95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0559e481e0c544cfb875ac477e5a1d0a. Latest consensus state: current_term: 1 leader_uuid: "0559e481e0c544cfb875ac477e5a1d0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0559e481e0c544cfb875ac477e5a1d0a" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 37915 } } peers { permanent_uuid: "6bdf138dce19469eb482f094cb9b882a" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 34023 } } peers { permanent_uuid: "dbdf9aa2862a4a9093a60043d7f07d95" member_type: VOTER last_known_addr { host: "127.29.234.60" port: 41183 } } }
I20260924 16:18:08.268168 21647 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbdf9aa2862a4a9093a60043d7f07d95 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:18:08.268435 21687 mvcc.cc:205] Tried to move back new op lower bound from 7332932355014475776 to 7332932354609471488. Current Snapshot: MvccSnapshot[applied={T|T < 7332932355014475776}]
I20260924 16:18:08.269809 21666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0559e481e0c544cfb875ac477e5a1d0a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0559e481e0c544cfb875ac477e5a1d0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0559e481e0c544cfb875ac477e5a1d0a" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 37915 } } peers { permanent_uuid: "6bdf138dce19469eb482f094cb9b882a" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 34023 } } peers { permanent_uuid: "dbdf9aa2862a4a9093a60043d7f07d95" member_type: VOTER last_known_addr { host: "127.29.234.60" port: 41183 } } }
I20260924 16:18:08.270088 21666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0559e481e0c544cfb875ac477e5a1d0a [sys.catalog]: This master's current role is: LEADER
I20260924 16:18:08.273258 21691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bdf138dce19469eb482f094cb9b882a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0559e481e0c544cfb875ac477e5a1d0a" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 37915 } } peers { permanent_uuid: "6bdf138dce19469eb482f094cb9b882a" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 34023 } } peers { permanent_uuid: "dbdf9aa2862a4a9093a60043d7f07d95" member_type: VOTER last_known_addr { host: "127.29.234.60" port: 41183 } } }
I20260924 16:18:08.275612 21691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bdf138dce19469eb482f094cb9b882a [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:18:08.275485 21564 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6bdf138dce19469eb482f094cb9b882a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 16:18:08.276321 21538 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6bdf138dce19469eb482f094cb9b882a [term 0 FOLLOWER]: Advancing to term 1
I20260924 16:18:08.275545 21539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0559e481e0c544cfb875ac477e5a1d0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6bdf138dce19469eb482f094cb9b882a"
I20260924 16:18:08.285199 21538 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6bdf138dce19469eb482f094cb9b882a [term 1 FOLLOWER]: Refusing update from remote peer 0559e481e0c544cfb875ac477e5a1d0a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 16:18:08.289598 21539 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6bdf138dce19469eb482f094cb9b882a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0559e481e0c544cfb875ac477e5a1d0a in term 1.
I20260924 16:18:08.292147 21662 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0559e481e0c544cfb875ac477e5a1d0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bdf138dce19469eb482f094cb9b882a" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 34023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 16:18:08.293799 21676 catalog_manager.cc:1347] Generated new cluster ID: 0e32661e9c3845cab8d7eac2bcb90a28
I20260924 16:18:08.293934 21676 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 16:18:08.307951 21647 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbdf9aa2862a4a9093a60043d7f07d95 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0559e481e0c544cfb875ac477e5a1d0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0559e481e0c544cfb875ac477e5a1d0a" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 37915 } } peers { permanent_uuid: "6bdf138dce19469eb482f094cb9b882a" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 34023 } } peers { permanent_uuid: "dbdf9aa2862a4a9093a60043d7f07d95" member_type: VOTER last_known_addr { host: "127.29.234.60" port: 41183 } } }
I20260924 16:18:08.308228 21647 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbdf9aa2862a4a9093a60043d7f07d95 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:18:08.322651 21676 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 16:18:08.324756 21676 catalog_manager.cc:1504] Loading token signing keys...
I20260924 16:18:08.329847 21691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bdf138dce19469eb482f094cb9b882a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0559e481e0c544cfb875ac477e5a1d0a. Latest consensus state: current_term: 1 leader_uuid: "0559e481e0c544cfb875ac477e5a1d0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0559e481e0c544cfb875ac477e5a1d0a" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 37915 } } peers { permanent_uuid: "6bdf138dce19469eb482f094cb9b882a" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 34023 } } peers { permanent_uuid: "dbdf9aa2862a4a9093a60043d7f07d95" member_type: VOTER last_known_addr { host: "127.29.234.60" port: 41183 } } }
I20260924 16:18:08.330191 21691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bdf138dce19469eb482f094cb9b882a [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:18:08.330111 21666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0559e481e0c544cfb875ac477e5a1d0a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0559e481e0c544cfb875ac477e5a1d0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0559e481e0c544cfb875ac477e5a1d0a" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 37915 } } peers { permanent_uuid: "6bdf138dce19469eb482f094cb9b882a" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 34023 } } peers { permanent_uuid: "dbdf9aa2862a4a9093a60043d7f07d95" member_type: VOTER last_known_addr { host: "127.29.234.60" port: 41183 } } }
I20260924 16:18:08.330407 21666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0559e481e0c544cfb875ac477e5a1d0a [sys.catalog]: This master's current role is: LEADER
I20260924 16:18:08.332674 21700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bdf138dce19469eb482f094cb9b882a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0559e481e0c544cfb875ac477e5a1d0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0559e481e0c544cfb875ac477e5a1d0a" member_type: VOTER last_known_addr { host: "127.29.234.62" port: 37915 } } peers { permanent_uuid: "6bdf138dce19469eb482f094cb9b882a" member_type: VOTER last_known_addr { host: "127.29.234.61" port: 34023 } } peers { permanent_uuid: "dbdf9aa2862a4a9093a60043d7f07d95" member_type: VOTER last_known_addr { host: "127.29.234.60" port: 41183 } } }
I20260924 16:18:08.340246 21700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bdf138dce19469eb482f094cb9b882a [sys.catalog]: This master's current role is: FOLLOWER
I20260924 16:18:08.333300 21696 mvcc.cc:205] Tried to move back new op lower bound from 7332932355014475776 to 7332932354609471488. Current Snapshot: MvccSnapshot[applied={T|T < 7332932355014475776 or (T in {7332932355014475776})}]
I20260924 16:18:08.358330 21676 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0559e481e0c544cfb875ac477e5a1d0a: Generated new TSK 0
I20260924 16:18:08.359568 21676 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 16:18:08.443089 21710 catalog_manager.cc:1259] Loaded cluster ID: 0e32661e9c3845cab8d7eac2bcb90a28
I20260924 16:18:08.443235 21710 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6bdf138dce19469eb482f094cb9b882a: loading cluster ID for follower catalog manager: success
I20260924 16:18:08.460037 21710 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6bdf138dce19469eb482f094cb9b882a: acquiring CA information for follower catalog manager: success
I20260924 16:18:08.464257 21710 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6bdf138dce19469eb482f094cb9b882a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 16:18:08.514551 21568 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:18:08.487992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56558 (local address 127.29.234.61:34023)
0924 16:18:08.488221 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0924 16:18:08.488228 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:18:08.510277 (+ 22049us) server_negotiation.cc:407] Connection header received
0924 16:18:08.511641 (+ 1364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:18:08.511648 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:18:08.511767 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:18:08.511846 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:18:08.514297 (+ 2451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56558: BlockingRecv error: recv got EOF from 127.0.0.1:56558 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260924 16:18:08.516052 21681 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:18:08.494668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35550 (local address 127.29.234.60:41183)
0924 16:18:08.494954 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0924 16:18:08.494964 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 16:18:08.511157 (+ 16193us) server_negotiation.cc:407] Connection header received
0924 16:18:08.512891 (+ 1734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:18:08.512896 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:18:08.512984 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:18:08.513057 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:18:08.515820 (+ 2763us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35550: BlockingRecv error: recv got EOF from 127.0.0.1:35550 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260924 16:18:08.518460 21645 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:18:08.508187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50100 (local address 127.29.234.62:37915)
0924 16:18:08.508405 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0924 16:18:08.508414 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 16:18:08.513837 (+ 5423us) server_negotiation.cc:407] Connection header received
0924 16:18:08.516849 (+ 3012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:18:08.516859 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:18:08.516973 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:18:08.517046 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:18:08.518189 (+ 1143us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50100: BlockingRecv error: recv got EOF from 127.0.0.1:50100 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
I20260924 16:18:08.567027 21568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:08.547621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56564 (local address 127.29.234.61:34023)
0924 16:18:08.547838 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 16:18:08.547844 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:18:08.558561 (+ 10717us) server_negotiation.cc:407] Connection header received
0924 16:18:08.559254 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:18:08.559261 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:18:08.559460 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:18:08.559598 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:18:08.560365 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:08.561542 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:08.563631 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:08.564348 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:08.564457 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:18:08.565244 (+ 787us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:18:08.565977 (+ 733us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:18:08.566114 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 16:18:08.566567 (+ 453us) server_negotiation.cc:299] Negotiation successful
0924 16:18:08.566678 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
W20260924 16:18:08.613914 21681 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:18:08.600518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35556 (local address 127.29.234.60:41183)
0924 16:18:08.600724 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 16:18:08.600730 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:18:08.611465 (+ 10735us) server_negotiation.cc:407] Connection header received
0924 16:18:08.612082 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:18:08.612089 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:18:08.612228 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:18:08.612315 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:18:08.613568 (+ 1253us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35556: BlockingRecv error: recv got EOF from 127.0.0.1:35556 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 16:18:08.668169 21568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:08.650057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56574 (local address 127.29.234.61:34023)
0924 16:18:08.650271 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 16:18:08.650278 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:18:08.660991 (+ 10713us) server_negotiation.cc:407] Connection header received
0924 16:18:08.661554 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:18:08.661561 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:18:08.661686 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:18:08.661775 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:18:08.662483 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:08.663653 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:08.665779 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:08.666525 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:08.666628 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:18:08.667272 (+ 644us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:18:08.667437 (+ 165us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:18:08.667497 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 16:18:08.667733 (+ 236us) server_negotiation.cc:299] Negotiation successful
0924 16:18:08.667831 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260924 16:18:08.722399 21681 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:18:08.709513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35564 (local address 127.29.234.60:41183)
0924 16:18:08.709729 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0924 16:18:08.709736 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:18:08.720233 (+ 10497us) server_negotiation.cc:407] Connection header received
0924 16:18:08.720775 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:18:08.720782 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:18:08.720881 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:18:08.720964 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:18:08.722149 (+ 1185us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35564: BlockingRecv error: recv got EOF from 127.0.0.1:35564 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 16:18:08.777515 21568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:08.759524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56586 (local address 127.29.234.61:34023)
0924 16:18:08.759720 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 16:18:08.759726 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:18:08.770404 (+ 10678us) server_negotiation.cc:407] Connection header received
0924 16:18:08.770940 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:18:08.770947 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:18:08.771042 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:18:08.771108 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:18:08.771830 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:08.773061 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:08.775171 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:08.775878 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:08.775980 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:18:08.776605 (+ 625us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:18:08.776759 (+ 154us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:18:08.776822 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 16:18:08.777049 (+ 227us) server_negotiation.cc:299] Negotiation successful
0924 16:18:08.777146 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260924 16:18:08.809914 21681 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:18:08.807914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35576 (local address 127.29.234.60:41183)
0924 16:18:08.808142 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0924 16:18:08.808148 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:18:08.808171 (+ 23us) server_negotiation.cc:407] Connection header received
0924 16:18:08.808431 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:18:08.808438 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:18:08.808556 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:18:08.808651 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:18:08.809673 (+ 1022us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35576: BlockingRecv error: recv got EOF from 127.0.0.1:35576 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
I20260924 16:18:08.824491 21568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:08.817759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56600 (local address 127.29.234.61:34023)
0924 16:18:08.817998 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0924 16:18:08.818006 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:18:08.818428 (+ 422us) server_negotiation.cc:407] Connection header received
0924 16:18:08.818619 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:18:08.818625 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:18:08.818727 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:18:08.818830 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:18:08.819268 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:08.820392 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:08.822379 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:08.823107 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:08.823236 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:18:08.823693 (+ 457us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:18:08.823843 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:18:08.823931 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 16:18:08.824081 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 16:18:08.824191 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
W20260924 16:18:08.834136 21681 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:18:08.831521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35588 (local address 127.29.234.60:41183)
0924 16:18:08.831791 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0924 16:18:08.831797 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:18:08.832366 (+ 569us) server_negotiation.cc:407] Connection header received
0924 16:18:08.832636 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:18:08.832643 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:18:08.832749 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:18:08.832826 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:18:08.833890 (+ 1064us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35588: BlockingRecv error: recv got EOF from 127.0.0.1:35588 (error 108)
Metrics: {"server-negotiator.queue_time_us":145}
I20260924 16:18:08.851166 21568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:08.845322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56604 (local address 127.29.234.61:34023)
0924 16:18:08.845552 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0924 16:18:08.845558 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:18:08.845980 (+ 422us) server_negotiation.cc:407] Connection header received
0924 16:18:08.846136 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:18:08.846141 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:18:08.846242 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:18:08.846352 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:18:08.846707 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:08.847797 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:08.849110 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:08.849795 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:08.849929 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:18:08.850377 (+ 448us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:18:08.850525 (+ 148us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:18:08.850615 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 16:18:08.850778 (+ 163us) server_negotiation.cc:299] Negotiation successful
0924 16:18:08.850859 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
W20260924 16:18:08.859007 21681 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:18:08.856929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35592 (local address 127.29.234.60:41183)
0924 16:18:08.857121 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 16:18:08.857127 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:18:08.857595 (+ 468us) server_negotiation.cc:407] Connection header received
0924 16:18:08.857766 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:18:08.857770 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:18:08.857842 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:18:08.857924 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:18:08.858768 (+ 844us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35592: BlockingRecv error: recv got EOF from 127.0.0.1:35592 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 16:18:08.885047 21568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:08.878962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56610 (local address 127.29.234.61:34023)
0924 16:18:08.879178 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0924 16:18:08.879185 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:18:08.879614 (+ 429us) server_negotiation.cc:407] Connection header received
0924 16:18:08.879807 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:18:08.879813 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:18:08.879913 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:18:08.880034 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:18:08.880430 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:08.881681 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:08.882991 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:08.883668 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:08.883808 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:18:08.884262 (+ 454us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:18:08.884402 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:18:08.884498 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 16:18:08.884691 (+ 193us) server_negotiation.cc:299] Negotiation successful
0924 16:18:08.884787 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260924 16:18:08.893730 21681 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:18:08.891689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35604 (local address 127.29.234.60:41183)
0924 16:18:08.891891 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 16:18:08.891898 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:18:08.892333 (+ 435us) server_negotiation.cc:407] Connection header received
0924 16:18:08.892454 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:18:08.892460 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:18:08.892563 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:18:08.892677 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:18:08.893485 (+ 808us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35604: BlockingRecv error: recv got EOF from 127.0.0.1:35604 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 16:18:08.938563 21568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:08.931424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56616 (local address 127.29.234.61:34023)
0924 16:18:08.931610 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 16:18:08.931616 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:18:08.932185 (+ 569us) server_negotiation.cc:407] Connection header received
0924 16:18:08.932413 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:18:08.932419 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:18:08.932540 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:18:08.932656 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:18:08.933184 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:08.934387 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:08.936326 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:08.937043 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:08.937198 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:18:08.937705 (+ 507us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:18:08.937848 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:18:08.937949 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 16:18:08.938124 (+ 175us) server_negotiation.cc:299] Negotiation successful
0924 16:18:08.938220 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260924 16:18:08.949951 21681 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:18:08.947659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35614 (local address 127.29.234.60:41183)
0924 16:18:08.947850 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 16:18:08.947857 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 16:18:08.948407 (+ 550us) server_negotiation.cc:407] Connection header received
0924 16:18:08.948610 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:18:08.948616 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:18:08.948721 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:18:08.948822 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:18:08.949720 (+ 898us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35614: BlockingRecv error: recv got EOF from 127.0.0.1:35614 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 16:18:09.023627 21568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:09.018132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56624 (local address 127.29.234.61:34023)
0924 16:18:09.018351 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 16:18:09.018359 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:18:09.018803 (+ 444us) server_negotiation.cc:407] Connection header received
0924 16:18:09.019064 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:18:09.019068 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:18:09.019139 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:18:09.019243 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:18:09.019712 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:09.020502 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:09.021881 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:09.022326 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:09.022470 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:18:09.022885 (+ 415us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:18:09.023001 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:18:09.023101 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 16:18:09.023257 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 16:18:09.023354 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260924 16:18:09.031579 21681 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 16:18:09.029487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35616 (local address 127.29.234.60:41183)
0924 16:18:09.029758 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0924 16:18:09.029766 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:18:09.030063 (+ 297us) server_negotiation.cc:407] Connection header received
0924 16:18:09.030296 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:18:09.030302 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:18:09.030398 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:18:09.030515 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 16:18:09.031355 (+ 840us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35616: BlockingRecv error: recv got EOF from 127.0.0.1:35616 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
I20260924 16:18:09.094893 21659 catalog_manager.cc:1259] Loaded cluster ID: 0e32661e9c3845cab8d7eac2bcb90a28
I20260924 16:18:09.095001 21659 catalog_manager.cc:1552] T 00000000000000000000000000000000 P dbdf9aa2862a4a9093a60043d7f07d95: loading cluster ID for follower catalog manager: success
I20260924 16:18:09.097774 21659 catalog_manager.cc:1574] T 00000000000000000000000000000000 P dbdf9aa2862a4a9093a60043d7f07d95: acquiring CA information for follower catalog manager: success
I20260924 16:18:09.099699 21659 catalog_manager.cc:1602] T 00000000000000000000000000000000 P dbdf9aa2862a4a9093a60043d7f07d95: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 16:18:09.172298 21568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:09.167292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56636 (local address 127.29.234.61:34023)
0924 16:18:09.167504 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0924 16:18:09.167510 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:18:09.167864 (+ 354us) server_negotiation.cc:407] Connection header received
0924 16:18:09.168019 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:18:09.168023 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:18:09.168097 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:18:09.168189 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:18:09.168587 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:09.169299 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:09.170652 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:09.171102 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:09.171198 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:18:09.171613 (+ 415us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:18:09.171743 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:18:09.171809 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 16:18:09.171993 (+ 184us) server_negotiation.cc:299] Negotiation successful
0924 16:18:09.172059 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260924 16:18:09.183749 21681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:09.178082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35630 (local address 127.29.234.60:41183)
0924 16:18:09.178317 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0924 16:18:09.178323 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 16:18:09.178568 (+ 245us) server_negotiation.cc:407] Connection header received
0924 16:18:09.178701 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:18:09.178705 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:18:09.178863 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:18:09.178963 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:18:09.179306 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:09.180061 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:09.181339 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:09.181822 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:09.181921 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:18:09.182410 (+ 489us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:18:09.183072 (+ 662us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:18:09.183199 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 16:18:09.183329 (+ 130us) server_negotiation.cc:299] Negotiation successful
0924 16:18:09.183401 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260924 16:18:09.201077 21681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:09.190170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35638 (local address 127.29.234.60:41183)
0924 16:18:09.190347 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 16:18:09.190352 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 16:18:09.194042 (+ 3690us) server_negotiation.cc:407] Connection header received
0924 16:18:09.194316 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:18:09.194325 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:18:09.194470 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:18:09.194585 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:18:09.195572 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:09.197015 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:09.199294 (+ 2279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:09.199734 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:09.199881 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:18:09.200312 (+ 431us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:18:09.200431 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:18:09.200533 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 16:18:09.200680 (+ 147us) server_negotiation.cc:299] Negotiation successful
0924 16:18:09.200764 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 16:18:09.206602 21568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:09.190331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56640 (local address 127.29.234.61:34023)
0924 16:18:09.190552 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 16:18:09.190580 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0924 16:18:09.198180 (+ 7600us) server_negotiation.cc:407] Connection header received
0924 16:18:09.198459 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:18:09.198466 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:18:09.198559 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:18:09.198647 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:18:09.200674 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:09.202112 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:09.204552 (+ 2440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:09.205091 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:09.205233 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:18:09.205740 (+ 507us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:18:09.205878 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:18:09.205976 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 16:18:09.206206 (+ 230us) server_negotiation.cc:299] Negotiation successful
0924 16:18:09.206308 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 16:18:09.208983 21817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 16:18:09.190068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50114 (local address 127.29.234.62:37915)
0924 16:18:09.203008 (+ 12940us) server_negotiation.cc:206] Beginning negotiation
0924 16:18:09.203016 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 16:18:09.203040 (+ 24us) server_negotiation.cc:407] Connection header received
0924 16:18:09.203165 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 16:18:09.203170 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 16:18:09.203419 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 16:18:09.203634 (+ 215us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 16:18:09.203985 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:09.204942 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:09.206873 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 16:18:09.207260 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 16:18:09.207404 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 16:18:09.207899 (+ 495us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 16:18:09.208334 (+ 435us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 16:18:09.208461 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 16:18:09.208597 (+ 136us) server_negotiation.cc:299] Negotiation successful
0924 16:18:09.208687 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12801,"thread_start_us":13672,"threads_started":1}
I20260924 16:18:09.230687 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 21419
I20260924 16:18:09.231931 21482 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:18:09.344885 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 21419
I20260924 16:18:09.366823 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 21494
I20260924 16:18:09.368021 21559 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:18:09.479395 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 21494
I20260924 16:18:09.500619 30632 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQurfSc/build/asan/bin/kudu pid 21571
I20260924 16:18:09.501839 21634 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 16:18:09.611101 30632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQurfSc/build/asan/bin/kudu with pid 21571
2026-09-24T16:18:09Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4537 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9050 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (366346 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST